[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:45 dist-test-slave-515x krb5kdc[8698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:46 dist-test-slave-515x krb5kdc[8698](info): set up 2 sockets
Sep 28 05:22:46 dist-test-slave-515x krb5kdc[8698](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 05:22:46.898692 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:22:47 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:22:47.056916 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38411
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:47.365227 8711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:47.365522 8711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:47.365685 8711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:47.372946 8711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:47.373044 8711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:47.373116 8711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:47.373188 8711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:47.373246 8711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:47.382817 8711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:47.384011 8711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:47.385270 8711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:47.396293 8717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:47.396384 8716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:47.396354 8719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:47.396687 8711 server_base.cc:1034] running on GCE node
I20260928 05:22:47.397666 8711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:47.398988 8711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:47.400184 8711 hybrid_clock.cc:648] HybridClock initialized: now 1790572967400123 us; error 53 us; skew 500 ppm
I20260928 05:22:47.400498 8711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:47 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:47.411923 8711 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:22:47.413512 8711 webserver.cc:460] Webserver started at http://127.8.124.254:35791/ using document root <none> and password file <none>
I20260928 05:22:47.414240 8711 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:47.414358 8711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:47.414638 8711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:47.417572 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8"
format_stamp: "Formatted at 2026-09-28 05:22:47 on dist-test-slave-515x"
server_key: "c83c7fb42062ddecd582d53b11d73a02"
server_key_iv: "dba01db37aff62ee5db48aee0462bd79"
server_key_version: "encryptionkey@0"
I20260928 05:22:47.418306 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8"
format_stamp: "Formatted at 2026-09-28 05:22:47 on dist-test-slave-515x"
server_key: "c83c7fb42062ddecd582d53b11d73a02"
server_key_iv: "dba01db37aff62ee5db48aee0462bd79"
server_key_version: "encryptionkey@0"
I20260928 05:22:47.423135 8711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:22:47.426412 8726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:47.427664 8711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:47.427814 8711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8"
format_stamp: "Formatted at 2026-09-28 05:22:47 on dist-test-slave-515x"
server_key: "c83c7fb42062ddecd582d53b11d73a02"
server_key_iv: "dba01db37aff62ee5db48aee0462bd79"
server_key_version: "encryptionkey@0"
I20260928 05:22:47.427973 8711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:47.444835 8711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:47.448532 8711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:47.448783 8711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:47.464823 8711 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:38411
I20260928 05:22:47.464841 8777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:38411 every 8 connection(s)
I20260928 05:22:47.466094 8711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:47.469492 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:47.471635 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 8711
I20260928 05:22:47.471890 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:22:47.473649 8691 external_mini_cluster.cc:1428] Setting key e216559e0a48f7c6ffa8ff113bfd1028
I20260928 05:22:47.480134 8778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8: Bootstrap starting.
I20260928 05:22:47.482990 8778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:47.483958 8778 log.cc:824] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:47.486754 8778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8: No bootstrap required, opened a new log
I20260928 05:22:47.493885 8778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } }
I20260928 05:22:47.494222 8778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:47.494314 8778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3df7ad89b84fb7b767f86d12dcfbe8, State: Initialized, Role: FOLLOWER
I20260928 05:22:47.494838 8778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } }
I20260928 05:22:47.495059 8778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:47.495173 8778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:47.495329 8778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:47.497447 8778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } }
I20260928 05:22:47.497900 8778 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3df7ad89b84fb7b767f86d12dcfbe8; no voters:
I20260928 05:22:47.498414 8778 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:47.498637 8783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 05:22:47 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:47.500228 8778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:47.505529 8783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3df7ad89b84fb7b767f86d12dcfbe8, State: Running, Role: LEADER
I20260928 05:22:47.506194 8783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } }
I20260928 05:22:47.508916 8784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } } }
I20260928 05:22:47.509269 8784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:47.513396 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:47.477260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.8.124.254:38411)
0928 05:22:47.478039 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:47.478058 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:47.479161 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 05:22:47.487736 (+ 8575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:47.487758 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:47.488148 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:47.488544 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:47.491374 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:47.492692 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:47.494393 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:47.495144 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:47.495318 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:47.501542 (+ 6224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:47.501554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:47.501574 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:47.501651 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:47.505726 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:47.506685 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:47.506693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:47.506700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:47.506906 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:47.507515 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:47.507521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:47.507525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:47.508279 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:47.511345 (+ 3066us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:47.511614 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 05:22:47.512048 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260928 05:22:47.515671 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3df7ad89b84fb7b767f86d12dcfbe8. Latest consensus state: current_term: 1 leader_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3df7ad89b84fb7b767f86d12dcfbe8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38411 } } }
I20260928 05:22:47.515993 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:47.520254 8793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:47.524852 8793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:47.534152 8793 catalog_manager.cc:1347] Generated new cluster ID: 1fd2cec5c7cc49b38fb81498e2e3db95
I20260928 05:22:47.534247 8793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:47.592404 8793 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:47.593732 8793 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:47.603659 8793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a3df7ad89b84fb7b767f86d12dcfbe8: Generated new TSK 0
I20260928 05:22:47.604708 8793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:22:47.655234 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:47.960215 8805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:47.960484 8805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:47.960618 8805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:47.967754 8805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:47.967877 8805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:47.968030 8805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:22:47.978003 8805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:47.979236 8805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:47.980486 8805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:47.992188 8810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:47.992199 8811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:47.992482 8813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:47.993146 8805 server_base.cc:1034] running on GCE node
I20260928 05:22:47.993678 8805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:47.994488 8805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:47.995690 8805 hybrid_clock.cc:648] HybridClock initialized: now 1790572967995629 us; error 52 us; skew 500 ppm
I20260928 05:22:47.995980 8805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:47 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:47.998868 8805 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:22:48.000226 8805 webserver.cc:460] Webserver started at http://127.8.124.193:36983/ using document root <none> and password file <none>
I20260928 05:22:48.000839 8805 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:48.000943 8805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:48.001253 8805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:48.004069 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "1a5ccc3e3ebe495e86f1aa0e61ae7bed"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "9e205d161db5b0930f3835b89736827b"
server_key_iv: "898177e63eb480fef61adca043126898"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.004809 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "1a5ccc3e3ebe495e86f1aa0e61ae7bed"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "9e205d161db5b0930f3835b89736827b"
server_key_iv: "898177e63eb480fef61adca043126898"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.010087 8805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:22:48.013309 8820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.014622 8805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:48.014752 8805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "1a5ccc3e3ebe495e86f1aa0e61ae7bed"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "9e205d161db5b0930f3835b89736827b"
server_key_iv: "898177e63eb480fef61adca043126898"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.014919 8805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:48.029282 8805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:48.032980 8805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:48.033221 8805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:48.034096 8805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:48.035255 8805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:48.035322 8805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.035444 8805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:48.035490 8805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.062008 8805 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:43895
I20260928 05:22:48.062042 8932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:43895 every 8 connection(s)
I20260928 05:22:48.063189 8805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:22:48.070327 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 8805
I20260928 05:22:48.070513 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:22:48.071067 8691 external_mini_cluster.cc:1428] Setting key b40a773c379f9ab925121f92bd1ca851
Sep 28 05:22:48 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:48.081240 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.065567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:34919 (local address 127.8.124.254:38411)
0928 05:22:48.065989 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:48.065993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:48.066903 (+ 910us) server_negotiation.cc:407] Connection header received
0928 05:22:48.068612 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:48.068618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:48.068694 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:48.068785 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:48.070286 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.071029 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.072505 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.072903 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.073006 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.076129 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:48.076134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:48.076137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:48.076166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:48.078550 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.079195 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.079199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.079201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.079311 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.079692 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.079697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.079699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.079948 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:48.080081 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:48.080708 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 05:22:48.080917 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260928 05:22:48.082437 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.066083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38411 (local address 127.8.124.193:34919)
0928 05:22:48.066754 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:48.066791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:48.068357 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:48.069021 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:48.069029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:48.069528 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:48.070124 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.070143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.071174 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.071179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.072371 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.072382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.073093 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.073100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.073313 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.073932 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:48.073952 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:48.075938 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:48.078670 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.078679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.078695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.079073 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.079406 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.079411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.079415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.079584 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.080063 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:48.080070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:48.080377 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 05:22:48.080594 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 05:22:48.081058 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":251,"threads_started":1}
I20260928 05:22:48.083868 8933 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38411
I20260928 05:22:48.084146 8933 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:22:48.084862 8933 heartbeater.cc:511] Master 127.8.124.254:38411 requested a full tablet report, sending...
I20260928 05:22:48.086858 8743 ts_manager.cc:194] Registered new tserver with Master: 1a5ccc3e3ebe495e86f1aa0e61ae7bed (127.8.124.193:43895)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:22:48.088780 8743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:34919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:22:48.106343 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:48.412295 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:48.412565 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:48.412698 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:48.419795 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:48.419893 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:48.420040 8939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:22:48.429422 8939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:48.430634 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:48.431912 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:48.443706 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:48.443895 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:48.444156 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:48.444571 8939 server_base.cc:1034] running on GCE node
I20260928 05:22:48.445225 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:48.446143 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:48.447341 8939 hybrid_clock.cc:648] HybridClock initialized: now 1790572968447287 us; error 48 us; skew 500 ppm
I20260928 05:22:48.447629 8939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:48 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:48.450559 8939 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:22:48.451862 8939 webserver.cc:460] Webserver started at http://127.8.124.194:46335/ using document root <none> and password file <none>
I20260928 05:22:48.452543 8939 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:48.452651 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:48.452944 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:48.455745 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "4ebf4d072f5c46fca3ee3f109e89284d"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "b97e2b658f30c110e0ac164ec17274f8"
server_key_iv: "df8cdb69075c780e7773f544dea4030c"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.456434 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "4ebf4d072f5c46fca3ee3f109e89284d"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "b97e2b658f30c110e0ac164ec17274f8"
server_key_iv: "df8cdb69075c780e7773f544dea4030c"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.461158 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 05:22:48.464486 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.465863 8939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:22:48.466002 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "4ebf4d072f5c46fca3ee3f109e89284d"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "b97e2b658f30c110e0ac164ec17274f8"
server_key_iv: "df8cdb69075c780e7773f544dea4030c"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.466171 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:48.489382 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:48.493064 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:48.493300 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:48.494171 8939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:48.495373 8939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:48.495438 8939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.495523 8939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:48.495568 8939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.522567 8939 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:36131
I20260928 05:22:48.522619 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:36131 every 8 connection(s)
I20260928 05:22:48.523804 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:22:48.533681 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 8939
I20260928 05:22:48.533854 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:22:48.534240 8691 external_mini_cluster.cc:1428] Setting key 9354014fa51aeb3aca863c64eb585ed2
Sep 28 05:22:48 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:48.542964 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.526345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:48053 (local address 127.8.124.254:38411)
0928 05:22:48.526514 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:48.526519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:48.527625 (+ 1106us) server_negotiation.cc:407] Connection header received
0928 05:22:48.529406 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:48.529411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:48.529509 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:48.529637 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:48.531302 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.531999 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.533451 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.533907 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.534019 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.537866 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:48.537871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:48.537874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:48.537902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:48.540196 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.540842 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.540847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.540851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.540999 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.541355 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.541359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.541363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.541703 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:48.541837 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:48.542450 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 05:22:48.542658 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:22:48.543717 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.526799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38411 (local address 127.8.124.194:48053)
0928 05:22:48.527472 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:48.527516 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:48.529136 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:48.529870 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:48.529878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:48.530512 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:48.531149 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.531168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.532143 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.532147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.533326 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.533337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.534098 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.534105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.534326 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.534956 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:48.534982 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:48.537674 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:48.540339 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.540346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.540357 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.540739 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.541088 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.541091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.541093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.541269 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.541814 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:48.541820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:48.542104 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 05:22:48.542333 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 05:22:48.542680 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":243,"threads_started":1}
I20260928 05:22:48.545034 9067 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38411
I20260928 05:22:48.545357 9067 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:48.546034 9067 heartbeater.cc:511] Master 127.8.124.254:38411 requested a full tablet report, sending...
I20260928 05:22:48.547374 8743 ts_manager.cc:194] Registered new tserver with Master: 4ebf4d072f5c46fca3ee3f109e89284d (127.8.124.194:36131)
I20260928 05:22:48.548161 8743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:48053
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:22:48.568404 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:48.867722 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:48.867983 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:48.868101 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:48.875169 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:48.875269 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:48.875412 9072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:22:48.885157 9072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:48.886482 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:48.887771 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:48.900256 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:48.900295 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:48.900455 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:48.901348 9072 server_base.cc:1034] running on GCE node
I20260928 05:22:48.901950 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:48.902814 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:48.904011 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790572968903963 us; error 43 us; skew 500 ppm
I20260928 05:22:48.904313 9072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:48 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:48.907327 9072 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:22:48.908628 9072 webserver.cc:460] Webserver started at http://127.8.124.195:44177/ using document root <none> and password file <none>
I20260928 05:22:48.909288 9072 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:48.909399 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:48.909719 9072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:48.912597 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "a28ba72a30bb4515b47279e54d6339e5"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "8f2ccc35f6a05d8bb4cd3a83049ad1f8"
server_key_iv: "057f1511f6024cbdf45594ab5e421ba5"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.913303 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "a28ba72a30bb4515b47279e54d6339e5"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "8f2ccc35f6a05d8bb4cd3a83049ad1f8"
server_key_iv: "057f1511f6024cbdf45594ab5e421ba5"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.918179 9072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:22:48.921451 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.922794 9072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:48.922940 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "a28ba72a30bb4515b47279e54d6339e5"
format_stamp: "Formatted at 2026-09-28 05:22:48 on dist-test-slave-515x"
server_key: "8f2ccc35f6a05d8bb4cd3a83049ad1f8"
server_key_iv: "057f1511f6024cbdf45594ab5e421ba5"
server_key_version: "encryptionkey@0"
I20260928 05:22:48.923125 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:48.939787 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:48.943431 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:48.943683 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:48.944537 9072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:48.945788 9072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:48.945863 9072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.945990 9072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:48.946041 9072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:48.974412 9072 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:35993
I20260928 05:22:48.974455 9199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:35993 every 8 connection(s)
I20260928 05:22:48.975610 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:22:48.981551 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9072
I20260928 05:22:48.981727 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:22:48.982125 8691 external_mini_cluster.cc:1428] Setting key a506e61fdc8a77a19ee710a92eb0fbd2
Sep 28 05:22:48 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:48.993749 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.977926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:46811 (local address 127.8.124.254:38411)
0928 05:22:48.978099 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:48.978104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:48.979213 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 05:22:48.980923 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:48.980927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:48.980991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:48.981072 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:48.982637 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.983517 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.985269 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.985709 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.985813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.988895 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:48.988899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:48.988903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:48.988933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:48.991096 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.991726 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.991731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.991733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.991840 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:48.992203 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:48.992208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:48.992211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:48.992451 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:48.992582 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:48.993179 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 05:22:48.993375 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:22:48.994603 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:48.978402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38411 (local address 127.8.124.195:46811)
0928 05:22:48.979059 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:48.979096 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:48.980695 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:48.981300 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:48.981308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:48.981905 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:48.982479 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.982497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.983662 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.983666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.985139 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:48.985150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:48.985914 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:48.985920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:48.986124 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:48.986688 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:48.986714 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:48.988705 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:48.991224 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.991232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.991248 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.991618 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.991935 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:48.991939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:48.991943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:48.992105 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:48.992566 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:48.992573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:48.992870 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 05:22:48.993093 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 05:22:48.993570 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260928 05:22:48.995795 9200 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38411
I20260928 05:22:48.996097 9200 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:48.996946 9200 heartbeater.cc:511] Master 127.8.124.254:38411 requested a full tablet report, sending...
I20260928 05:22:48.998286 8743 ts_manager.cc:194] Registered new tserver with Master: a28ba72a30bb4515b47279e54d6339e5 (127.8.124.195:35993)
I20260928 05:22:48.999053 8743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:46811
I20260928 05:22:49.009825 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:49.049250 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:49.035591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.8.124.193:43895)
0928 05:22:49.036079 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:49.036086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:49.036216 (+ 130us) server_negotiation.cc:407] Connection header received
0928 05:22:49.036359 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:49.036365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:49.036576 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:49.036697 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:49.037985 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.038940 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.040108 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.040632 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.040723 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:49.044243 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:49.044257 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:49.044275 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:49.044319 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:49.046992 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.047543 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.047549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.047551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.047656 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.047941 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.047944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.047946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.048263 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:49.048486 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:49.048686 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 05:22:49.048908 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":227,"threads_started":1}
W20260928 05:22:49.050462 8848 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58796
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:49.090328 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:49.077971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.8.124.193:43895)
0928 05:22:49.078155 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:49.078163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:49.078483 (+ 320us) server_negotiation.cc:407] Connection header received
0928 05:22:49.078628 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:49.078632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:49.078704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:49.078793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:49.079974 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.080620 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.081793 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.082172 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.082261 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:49.085734 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:49.085740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:49.085743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:49.085776 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:49.088117 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.088709 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.088712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.088726 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.088831 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.089138 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.089141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.089143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.089390 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:49.089547 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:49.089919 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 05:22:49.090078 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:22:49.091626 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 8805
I20260928 05:22:49.091804 8933 heartbeater.cc:503] Master 127.8.124.254:38411 was elected leader, sending a full tablet report...
I20260928 05:22:49.103940 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:49.092763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.8.124.193:43895)
0928 05:22:49.092936 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:49.092941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:49.093013 (+ 72us) server_negotiation.cc:407] Connection header received
0928 05:22:49.093154 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:49.093159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:49.093226 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:49.093307 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:49.094496 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.095161 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.096347 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.096750 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.096838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:49.099015 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:49.099020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:49.099023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:49.099052 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:49.101483 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.102165 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.102206 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.102209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.102334 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.102708 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.102712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.102714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.102963 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:49.103140 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:49.103437 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 05:22:49.103586 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:22:49.104619 8928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:49.188023 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 8805
I20260928 05:22:49.200445 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 8939
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:22:49.215987 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:49.201301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.8.124.194:36131)
0928 05:22:49.201807 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:49.201814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:49.201840 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:22:49.201899 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:49.201905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:49.202144 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:49.202280 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:49.203456 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.204467 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.205743 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.206291 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.206407 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:49.210483 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:49.210494 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:49.210510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:49.210550 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:49.213168 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.213942 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.213950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.213953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.214079 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.214472 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.214475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.214477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.214825 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:49.215099 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:49.215391 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:22:49.215641 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":227,"threads_started":1}
I20260928 05:22:49.217144 9062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:49.296343 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 8939
I20260928 05:22:49.308355 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9072
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[8698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:22:49.323379 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:49.309343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.8.124.195:35993)
0928 05:22:49.309975 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:49.309982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:49.310009 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:22:49.310068 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:49.310074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:49.310301 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:49.310437 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:49.311535 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.312512 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.313835 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:49.314355 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:49.314445 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:49.318106 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:49.318116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:49.318131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:49.318162 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:49.320795 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.321502 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.321509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.321511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.321759 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:49.322044 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:49.322047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:49.322049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:49.322366 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:49.322576 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:49.322822 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 05:22:49.323013 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":338,"threads_started":1}
I20260928 05:22:49.324641 9195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:49.405393 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9072
I20260928 05:22:49.417773 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 8711
I20260928 05:22:49.418919 8773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:49.516870 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 8711
2026-09-28T05:22:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3611 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[9221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[9221](info): set up 2 sockets
Sep 28 05:22:49 dist-test-slave-515x krb5kdc[9221](info): commencing operation
krb5kdc: starting...
W20260928 05:22:50.509159 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.009s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:22:50 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:22:50.669028 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:35171
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:35171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:50.972958 9234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:50.973248 9234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:50.973335 9234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:50.980337 9234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:50.980427 9234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:50.980479 9234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:50.980531 9234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:50.980572 9234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:50.989933 9234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:35171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:50.991139 9234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:50.992410 9234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:51.002942 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:51.002966 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:51.003077 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:51.003638 9234 server_base.cc:1034] running on GCE node
I20260928 05:22:51.004474 9234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:51.005852 9234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:51.007056 9234 hybrid_clock.cc:648] HybridClock initialized: now 1790572971007010 us; error 39 us; skew 500 ppm
I20260928 05:22:51.007432 9234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:51 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:51.010432 9234 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:22:51.011866 9234 webserver.cc:460] Webserver started at http://127.8.124.254:45849/ using document root <none> and password file <none>
I20260928 05:22:51.012576 9234 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:51.012679 9234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:51.012961 9234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:51.015841 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "181229f00b26425c89f74dc346afacb2"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "40e8ad9e51bdde41dacfbbc54733954b"
server_key_iv: "f625594abd0a097bafbb915f578a9ec8"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.016537 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "181229f00b26425c89f74dc346afacb2"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "40e8ad9e51bdde41dacfbbc54733954b"
server_key_iv: "f625594abd0a097bafbb915f578a9ec8"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.021390 9234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:22:51.024552 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:51.025851 9234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:22:51.026001 9234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "181229f00b26425c89f74dc346afacb2"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "40e8ad9e51bdde41dacfbbc54733954b"
server_key_iv: "f625594abd0a097bafbb915f578a9ec8"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.026160 9234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:51.051910 9234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:51.055912 9234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:51.056157 9234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:51.072487 9234 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:35171
I20260928 05:22:51.072505 9300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:35171 every 8 connection(s)
I20260928 05:22:51.073830 9234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:51.077340 9301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:51.083746 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9234
I20260928 05:22:51.083976 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:22:51.084761 8691 external_mini_cluster.cc:1428] Setting key 6ac287b47b97f46bf0e591ef6d19bf61
I20260928 05:22:51.088163 9301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2: Bootstrap starting.
I20260928 05:22:51.091249 9301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:51.092234 9301 log.cc:824] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:51.094806 9301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2: No bootstrap required, opened a new log
Sep 28 05:22:51 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:51.102862 9301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } }
I20260928 05:22:51.103235 9301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:51.103343 9301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181229f00b26425c89f74dc346afacb2, State: Initialized, Role: FOLLOWER
I20260928 05:22:51.103988 9301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } }
I20260928 05:22:51.104202 9301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:51.104328 9301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:51.104506 9301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:51.106549 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:51.086882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.8.124.254:35171)
0928 05:22:51.087612 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:51.087624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:51.087663 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:22:51.089421 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:51.089447 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:51.089889 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:51.090245 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:51.091640 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.093017 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.094272 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.094915 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.095055 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:51.099419 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:51.099429 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:51.099448 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:51.099489 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:51.102446 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:51.103063 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:51.103070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:51.103076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:51.103245 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:51.103724 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:51.103730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:51.103733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:51.104327 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:51.104579 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:51.105056 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 05:22:51.105447 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260928 05:22:51.107095 9301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } }
I20260928 05:22:51.107558 9301 leader_election.cc:302] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181229f00b26425c89f74dc346afacb2; no voters:
I20260928 05:22:51.108101 9301 leader_election.cc:288] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:51.108256 9306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:51.109254 9306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [term 1 LEADER]: Becoming Leader. State: Replica: 181229f00b26425c89f74dc346afacb2, State: Running, Role: LEADER
I20260928 05:22:51.109854 9306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } }
I20260928 05:22:51.110261 9301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:51.112396 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181229f00b26425c89f74dc346afacb2. Latest consensus state: current_term: 1 leader_uuid: "181229f00b26425c89f74dc346afacb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } } }
I20260928 05:22:51.112579 9307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181229f00b26425c89f74dc346afacb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181229f00b26425c89f74dc346afacb2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35171 } } }
I20260928 05:22:51.112824 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:51.112866 9307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:51.124788 9315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:51.129995 9315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:51.140224 9315 catalog_manager.cc:1347] Generated new cluster ID: fc56bf1dbbff48b4b0fb0b9478ba4a63
I20260928 05:22:51.140336 9315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:51.155025 9315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:51.156891 9315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:51.164346 9315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181229f00b26425c89f74dc346afacb2: Generated new TSK 0
I20260928 05:22:51.165148 9315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:22:51.216771 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:51.523707 9327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:51.523993 9327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:51.524121 9327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:51.531549 9327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:51.531678 9327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:51.531837 9327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:22:51.541388 9327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:51.542631 9327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:51.543929 9327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:51.556126 9332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:51.556201 9335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:51.556162 9333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:51.556942 9327 server_base.cc:1034] running on GCE node
I20260928 05:22:51.557446 9327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:51.558331 9327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:51.559511 9327 hybrid_clock.cc:648] HybridClock initialized: now 1790572971559464 us; error 40 us; skew 500 ppm
I20260928 05:22:51.559801 9327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:51 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:51.562628 9327 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:22:51.563970 9327 webserver.cc:460] Webserver started at http://127.8.124.193:39839/ using document root <none> and password file <none>
I20260928 05:22:51.564599 9327 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:51.564702 9327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:51.564993 9327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:51.567857 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "d2e7d020166b469487feac488c238bf5"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "0f595c77efd6f27013fb4c981457f286"
server_key_iv: "7b8f491d1411a9420f41871ee1f92803"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.568584 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "d2e7d020166b469487feac488c238bf5"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "0f595c77efd6f27013fb4c981457f286"
server_key_iv: "7b8f491d1411a9420f41871ee1f92803"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.573475 9327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:22:51.576671 9342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:51.577939 9327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:22:51.578069 9327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "d2e7d020166b469487feac488c238bf5"
format_stamp: "Formatted at 2026-09-28 05:22:51 on dist-test-slave-515x"
server_key: "0f595c77efd6f27013fb4c981457f286"
server_key_iv: "7b8f491d1411a9420f41871ee1f92803"
server_key_version: "encryptionkey@0"
I20260928 05:22:51.578230 9327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:51.594352 9327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:51.597734 9327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:51.597975 9327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:51.598800 9327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:51.599977 9327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:51.600044 9327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:51.600154 9327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:51.600203 9327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:51.628552 9327 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:44449
I20260928 05:22:51.628592 9454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:44449 every 8 connection(s)
I20260928 05:22:51.629861 9327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:22:51.632791 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9327
I20260928 05:22:51.632977 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:22:51.633744 8691 external_mini_cluster.cc:1428] Setting key 2573765dc5fcd85a39d166b23e7dd8ac
Sep 28 05:22:51 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:22:51.650394 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:51.632232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:44703 (local address 127.8.124.254:35171)
0928 05:22:51.632650 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:51.632657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:51.636055 (+ 3398us) server_negotiation.cc:407] Connection header received
0928 05:22:51.636115 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:51.636120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:51.636199 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:51.636304 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:51.637785 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.638772 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.640291 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.640882 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.641002 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:51.644433 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:51.644438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:51.644442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:51.644474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:51.647106 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:51.648044 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:51.648048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:51.648050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:51.648166 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:51.648630 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:51.648633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:51.648635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:51.648898 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:51.649008 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:51.649838 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 05:22:51.650078 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":193,"threads_started":1}
I20260928 05:22:51.651275 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:51.632851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35171 (local address 127.8.124.193:44703)
0928 05:22:51.633676 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:51.633729 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:51.635694 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:51.636527 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:51.636536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:51.637029 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:51.637583 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:51.637633 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.638906 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.638910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:51.640147 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:51.640159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:51.641064 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:51.641069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:51.641268 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:51.641925 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:51.641948 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:51.644195 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:51.647274 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:51.647285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:51.647323 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:51.647918 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:51.648274 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:51.648279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:51.648283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:51.648527 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:51.649021 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:51.649029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:51.649457 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 05:22:51.649726 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 05:22:51.650152 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:22:51.652853 9455 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35171
I20260928 05:22:51.653138 9455 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:51.653926 9455 heartbeater.cc:511] Master 127.8.124.254:35171 requested a full tablet report, sending...
I20260928 05:22:51.656070 9266 ts_manager.cc:194] Registered new tserver with Master: d2e7d020166b469487feac488c238bf5 (127.8.124.193:44449)
I20260928 05:22:51.657467 9266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:44703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:22:51.672678 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:51.973194 9461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:51.973475 9461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:51.973636 9461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:51.980741 9461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:51.980849 9461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:51.980997 9461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:22:51.990439 9461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:51.991576 9461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:51.992792 9461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:52.004530 9466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:52.004513 9467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:52.004576 9469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:52.005294 9461 server_base.cc:1034] running on GCE node
I20260928 05:22:52.005884 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:52.006788 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:52.007987 9461 hybrid_clock.cc:648] HybridClock initialized: now 1790572972007942 us; error 51 us; skew 500 ppm
I20260928 05:22:52.008302 9461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:52.011360 9461 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:22:52.012848 9461 webserver.cc:460] Webserver started at http://127.8.124.194:42233/ using document root <none> and password file <none>
I20260928 05:22:52.013690 9461 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:52.013824 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:52.014168 9461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:52.018134 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "dba507306049476ea4bef69fa2227a6b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8b7b238b72b86bec42bae97cc3da934b"
server_key_iv: "75e7d6f327a9745e8a6bed0bee9500f6"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.019119 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "dba507306049476ea4bef69fa2227a6b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8b7b238b72b86bec42bae97cc3da934b"
server_key_iv: "75e7d6f327a9745e8a6bed0bee9500f6"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.025450 9461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 05:22:52.029704 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.031430 9461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:22:52.031582 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "dba507306049476ea4bef69fa2227a6b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8b7b238b72b86bec42bae97cc3da934b"
server_key_iv: "75e7d6f327a9745e8a6bed0bee9500f6"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.031790 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:52.069128 9461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:52.073340 9461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:52.073621 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:52.074542 9461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:52.075764 9461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:52.075838 9461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.075955 9461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:52.076005 9461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.108516 9461 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:39753
I20260928 05:22:52.108564 9588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:39753 every 8 connection(s)
I20260928 05:22:52.109814 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:22:52.117013 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9461
I20260928 05:22:52.117210 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:22:52.117761 8691 external_mini_cluster.cc:1428] Setting key a15109a1589241c66890c356e9f0b961
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:52.129225 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.112300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:36945 (local address 127.8.124.254:35171)
0928 05:22:52.112505 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.112512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.113695 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 05:22:52.115438 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.115443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.115510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.115595 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.117010 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.118089 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.119602 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.119996 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.120101 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.123635 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.123641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.123645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.123679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.126231 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.127001 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.127007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.127010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.127141 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.127547 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.127551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.127553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.127810 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.127953 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.128647 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.128872 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:22:52.130376 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.112850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35171 (local address 127.8.124.194:36945)
0928 05:22:52.113533 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.113570 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.115183 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.115783 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.115791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.116299 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:52.116848 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.116866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.118226 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.118230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.119461 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.119473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.120203 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.120210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.120439 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.121058 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:52.121080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:52.123389 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:52.126403 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:52.126413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:52.126435 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:52.126878 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:52.127246 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:52.127251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:52.127255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:52.127444 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:52.127933 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:52.127942 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:52.128303 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.128553 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.129115 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260928 05:22:52.131860 9589 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35171
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:22:52.132169 9589 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:52.132952 9589 heartbeater.cc:511] Master 127.8.124.254:35171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
I20260928 05:22:52.134613 9266 ts_manager.cc:194] Registered new tserver with Master: dba507306049476ea4bef69fa2227a6b (127.8.124.194:39753)
I20260928 05:22:52.135773 9266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:36945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:22:52.154167 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:52.456229 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:52.456574 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:52.456696 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:52.463780 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:52.463886 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:52.464036 9594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:22:52.473498 9594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35171
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:52.474792 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:52.476042 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:52.488304 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:52.488356 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:52.488369 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:52.488968 9594 server_base.cc:1034] running on GCE node
I20260928 05:22:52.489542 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:52.490479 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:52.491683 9594 hybrid_clock.cc:648] HybridClock initialized: now 1790572972491627 us; error 41 us; skew 500 ppm
I20260928 05:22:52.491986 9594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:52.499039 9594 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:22:52.500420 9594 webserver.cc:460] Webserver started at http://127.8.124.195:46615/ using document root <none> and password file <none>
I20260928 05:22:52.501056 9594 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:52.501156 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:52.501407 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:52.504158 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "0ec8aa2638cb446699a581bf090f712b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8a32ef0a0d11e79f7ae7da3aec4d0508"
server_key_iv: "9fa931d2c37153b9c3ed07731679aabe"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.504853 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "0ec8aa2638cb446699a581bf090f712b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8a32ef0a0d11e79f7ae7da3aec4d0508"
server_key_iv: "9fa931d2c37153b9c3ed07731679aabe"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.509690 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:22:52.513015 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.514323 9594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:52.514446 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "0ec8aa2638cb446699a581bf090f712b"
format_stamp: "Formatted at 2026-09-28 05:22:52 on dist-test-slave-515x"
server_key: "8a32ef0a0d11e79f7ae7da3aec4d0508"
server_key_iv: "9fa931d2c37153b9c3ed07731679aabe"
server_key_version: "encryptionkey@0"
I20260928 05:22:52.514599 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:52.528450 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:52.532611 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:52.532884 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:52.533835 9594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:22:52.535027 9594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:52.535094 9594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.535210 9594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:52.535261 9594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:52.563951 9594 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:36767
I20260928 05:22:52.563989 9721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:36767 every 8 connection(s)
I20260928 05:22:52.565218 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:22:52.568257 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9594
I20260928 05:22:52.568456 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:22:52.569005 8691 external_mini_cluster.cc:1428] Setting key a018c520273bcdb550cdf010c6672f22
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:52.583938 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.567614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:54741 (local address 127.8.124.254:35171)
0928 05:22:52.567772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.567777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.569218 (+ 1441us) server_negotiation.cc:407] Connection header received
0928 05:22:52.571254 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.571260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.571351 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.571437 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.572969 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.573744 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.575217 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.575617 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.575722 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.578866 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.578871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.578875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.578905 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.581073 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.581709 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.581715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.581718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.581863 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.582245 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.582250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.582254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.582538 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.582672 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.583410 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.583611 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:22:52.584837 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.568227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35171 (local address 127.8.124.195:54741)
0928 05:22:52.569053 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.569103 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.571004 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.571672 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.571681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.572269 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:52.572814 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.572832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.573894 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.573898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.575088 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.575099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.575816 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.575823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.576046 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.576650 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:52.576672 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:52.578670 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:52.581197 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:52.581205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:52.581221 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:52.581594 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:52.581957 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:52.581962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:52.581966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:52.582137 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:52.582670 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:52.582678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:52.583040 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.583302 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.583780 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":313,"threads_started":1}
I20260928 05:22:52.586053 9722 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35171
I20260928 05:22:52.586339 9722 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:52.586989 9722 heartbeater.cc:511] Master 127.8.124.254:35171 requested a full tablet report, sending...
I20260928 05:22:52.588213 9266 ts_manager.cc:194] Registered new tserver with Master: 0ec8aa2638cb446699a581bf090f712b (127.8.124.195:36767)
I20260928 05:22:52.588955 9266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:54741
I20260928 05:22:52.595851 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:22:52.613404 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.601856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.8.124.254:35171)
0928 05:22:52.602004 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.602008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.602486 (+ 478us) server_negotiation.cc:407] Connection header received
0928 05:22:52.602618 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.602622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.602697 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.602772 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.604063 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.604799 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.606010 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.606426 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.606517 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.608573 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.608578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.608581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.608611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.611062 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.611680 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.611684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.611686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.611839 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.612172 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.612176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.612178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.612474 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.612664 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.613018 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.613178 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:22:52.619499 9266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:22:52.626027 9266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:22:52.646576 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.637801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.8.124.195:36767)
0928 05:22:52.638751 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.638760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.638793 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:22:52.638886 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.638892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.639125 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.640027 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:52.640254 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.641786 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.644039 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.645435 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.645561 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.645820 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.646070 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.646241 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":630,"threads_started":1}
I20260928 05:22:52.646606 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.637543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36767 (local address 127.0.0.1:40020)
0928 05:22:52.638238 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.638291 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.638443 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.639410 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.639419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.639455 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:52.639835 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.639847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.641891 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.641896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.643898 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.643909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.645632 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.645639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.645844 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.645895 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.646021 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.646214 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":21,"thread_start_us":347,"threads_started":1}
I20260928 05:22:52.647508 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.638621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:39753 (local address 127.0.0.1:54504)
0928 05:22:52.639056 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.639071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.639162 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.640681 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.640686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.640710 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:52.641028 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.641039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.642819 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.642825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.644927 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.644943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.646697 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.646704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.646857 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.646888 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.646951 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.647077 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":27,"thread_start_us":215,"threads_started":1}
I20260928 05:22:52.647987 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.639125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:44449 (local address 127.0.0.1:59818)
0928 05:22:52.639613 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.639628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.639728 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.641669 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.641673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.641694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:52.642099 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.642109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.643837 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.643846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.645233 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.645244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.647230 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.647236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.647406 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.647435 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.647525 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.647639 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":19,"thread_start_us":655,"threads_started":1}
I20260928 05:22:52.648200 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.639608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.8.124.193:44449)
0928 05:22:52.641257 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.641264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.641284 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:22:52.641349 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.641353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.641487 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.641652 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:52.642274 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.643703 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.645375 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.647024 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.647149 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.647364 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.647759 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.647913 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":1388,"threads_started":1}
I20260928 05:22:52.648835 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.638888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.8.124.194:39753)
0928 05:22:52.640130 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.640137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.640162 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:22:52.640265 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.640272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.640493 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.640689 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:52.641202 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.642510 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.645117 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.646456 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.648179 (+ 1723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.648359 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.648436 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.648577 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":1012,"threads_started":1}
I20260928 05:22:52.651568 9524 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc (DEFAULT_TABLE table=test-table [id=4432a1cc865e4560a116373103f6d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:52.651437 9390 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc (DEFAULT_TABLE table=test-table [id=4432a1cc865e4560a116373103f6d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:52.651376 9657 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc (DEFAULT_TABLE table=test-table [id=4432a1cc865e4560a116373103f6d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:52.652674 9524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:52.652770 9390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:52.652776 9657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:52.660151 9455 heartbeater.cc:503] Master 127.8.124.254:35171 was elected leader, sending a full tablet report...
I20260928 05:22:52.663715 9740 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Bootstrap starting.
I20260928 05:22:52.667042 9740 tablet_bootstrap.cc:654] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:52.667850 9742 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Bootstrap starting.
I20260928 05:22:52.668255 9740 log.cc:824] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:52.669521 9741 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Bootstrap starting.
I20260928 05:22:52.671046 9740 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: No bootstrap required, opened a new log
I20260928 05:22:52.671283 9740 ts_tablet_manager.cc:1397] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:22:52.671347 9742 tablet_bootstrap.cc:654] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:52.672487 9742 log.cc:824] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:52.672727 9741 tablet_bootstrap.cc:654] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:52.673697 9741 log.cc:824] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:52.675218 9742 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: No bootstrap required, opened a new log
I20260928 05:22:52.675484 9742 ts_tablet_manager.cc:1397] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 05:22:52.676450 9741 tablet_bootstrap.cc:492] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: No bootstrap required, opened a new log
I20260928 05:22:52.676693 9741 ts_tablet_manager.cc:1397] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:22:52.678927 9740 raft_consensus.cc:348] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.679250 9740 raft_consensus.cc:374] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:52.679338 9740 raft_consensus.cc:729] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba507306049476ea4bef69fa2227a6b, State: Initialized, Role: FOLLOWER
I20260928 05:22:52.679852 9740 consensus_queue.cc:260] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.685629 9742 raft_consensus.cc:348] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.686038 9742 raft_consensus.cc:374] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:52.686153 9742 raft_consensus.cc:729] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec8aa2638cb446699a581bf090f712b, State: Initialized, Role: FOLLOWER
I20260928 05:22:52.686502 9741 raft_consensus.cc:348] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.686779 9589 heartbeater.cc:503] Master 127.8.124.254:35171 was elected leader, sending a full tablet report...
I20260928 05:22:52.686889 9741 raft_consensus.cc:374] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:52.686966 9741 raft_consensus.cc:729] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e7d020166b469487feac488c238bf5, State: Initialized, Role: FOLLOWER
I20260928 05:22:52.686818 9742 consensus_queue.cc:260] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.687461 9740 ts_tablet_manager.cc:1428] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
I20260928 05:22:52.687595 9741 consensus_queue.cc:260] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.689549 9741 ts_tablet_manager.cc:1428] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 05:22:52.690466 9722 heartbeater.cc:503] Master 127.8.124.254:35171 was elected leader, sending a full tablet report...
I20260928 05:22:52.692209 9742 ts_tablet_manager.cc:1428] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260928 05:22:52.744554 9748 raft_consensus.cc:482] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:52.744822 9748 raft_consensus.cc:504] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.746506 9748 leader_election.cc:288] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec8aa2638cb446699a581bf090f712b (127.8.124.195:36767), dba507306049476ea4bef69fa2227a6b (127.8.124.194:39753)
I20260928 05:22:52.752446 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.746908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:37705 (local address 127.8.124.195:36767)
0928 05:22:52.747125 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.747132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.747624 (+ 492us) server_negotiation.cc:407] Connection header received
0928 05:22:52.747709 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.747715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.747811 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.747900 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:52.748456 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.749172 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.750888 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.751614 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.751765 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.751839 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.752097 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.752217 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:22:52.752539 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.746851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36767 (local address 127.8.124.193:37705)
0928 05:22:52.747411 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.747427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.747542 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.747998 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.748003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.748029 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:52.748322 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.748332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.749301 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.749305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.750745 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.750756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.751803 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.751809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.751946 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.751970 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.752055 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.752179 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":289,"threads_started":1}
I20260928 05:22:52.752861 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.747720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:48299 (local address 127.8.124.194:39753)
0928 05:22:52.747922 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.747928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.748166 (+ 238us) server_negotiation.cc:407] Connection header received
0928 05:22:52.748324 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.748329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.748389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.748457 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:52.748932 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.749590 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.751235 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.752030 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.752164 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.752230 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.752500 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.752612 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:22:52.753541 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.747558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:39753 (local address 127.8.124.193:48299)
0928 05:22:52.748074 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:52.748089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:52.748205 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:52.748482 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:52.748486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:52.748508 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:52.748800 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.748808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.749728 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.749734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.751105 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:52.751116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.752208 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.752215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:52.752354 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.752376 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:22:52.752451 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:22:52.752651 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":248,"threads_started":1}
I20260928 05:22:52.753512 9677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd4130b5caa4ab5ab2807f3ab801dcc" candidate_uuid: "d2e7d020166b469487feac488c238bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8aa2638cb446699a581bf090f712b" is_pre_election: true
I20260928 05:22:52.754112 9677 raft_consensus.cc:2454] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2e7d020166b469487feac488c238bf5 in term 0.
I20260928 05:22:52.754468 9544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd4130b5caa4ab5ab2807f3ab801dcc" candidate_uuid: "d2e7d020166b469487feac488c238bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba507306049476ea4bef69fa2227a6b" is_pre_election: true
I20260928 05:22:52.754714 9343 leader_election.cc:302] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8aa2638cb446699a581bf090f712b, d2e7d020166b469487feac488c238bf5; no voters:
I20260928 05:22:52.754932 9544 raft_consensus.cc:2454] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2e7d020166b469487feac488c238bf5 in term 0.
I20260928 05:22:52.755072 9748 raft_consensus.cc:2781] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:22:52.755208 9748 raft_consensus.cc:482] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:22:52.755350 9748 raft_consensus.cc:3037] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:52.757400 9748 raft_consensus.cc:504] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.757928 9748 leader_election.cc:288] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec8aa2638cb446699a581bf090f712b (127.8.124.195:36767), dba507306049476ea4bef69fa2227a6b (127.8.124.194:39753)
I20260928 05:22:52.758342 9677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd4130b5caa4ab5ab2807f3ab801dcc" candidate_uuid: "d2e7d020166b469487feac488c238bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8aa2638cb446699a581bf090f712b"
I20260928 05:22:52.758381 9544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd4130b5caa4ab5ab2807f3ab801dcc" candidate_uuid: "d2e7d020166b469487feac488c238bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba507306049476ea4bef69fa2227a6b"
I20260928 05:22:52.758503 9677 raft_consensus.cc:3037] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:52.758526 9544 raft_consensus.cc:3037] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:52.760493 9677 raft_consensus.cc:2454] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e7d020166b469487feac488c238bf5 in term 1.
I20260928 05:22:52.760601 9544 raft_consensus.cc:2454] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e7d020166b469487feac488c238bf5 in term 1.
I20260928 05:22:52.760835 9343 leader_election.cc:302] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8aa2638cb446699a581bf090f712b, d2e7d020166b469487feac488c238bf5; no voters:
I20260928 05:22:52.761062 9748 raft_consensus.cc:2781] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:52.761459 9748 raft_consensus.cc:686] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [term 1 LEADER]: Becoming Leader. State: Replica: d2e7d020166b469487feac488c238bf5, State: Running, Role: LEADER
I20260928 05:22:52.761962 9748 consensus_queue.cc:237] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } }
I20260928 05:22:52.766587 9266 catalog_manager.cc:5495] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2e7d020166b469487feac488c238bf5 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "d2e7d020166b469487feac488c238bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec8aa2638cb446699a581bf090f712b" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba507306049476ea4bef69fa2227a6b" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e7d020166b469487feac488c238bf5" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 44449 } health_report { overall_health: HEALTHY } } }
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:52.809111 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.795720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.8.124.193:44449)
0928 05:22:52.795943 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.795950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.796279 (+ 329us) server_negotiation.cc:407] Connection header received
0928 05:22:52.796456 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.796459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.796519 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.796615 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.798088 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.798793 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.800015 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.800392 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.800497 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.804205 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.804220 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.804237 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.804271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.806835 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.807462 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.807469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.807472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.807617 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.807903 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.807906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.807908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.808230 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.808442 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.808648 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.808817 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 05:22:52.816677 9723 tablet.cc:2369] T 5fd4130b5caa4ab5ab2807f3ab801dcc P 0ec8aa2638cb446699a581bf090f712b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:52.849852 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.837789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.8.124.193:44449)
0928 05:22:52.838013 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.838020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.838293 (+ 273us) server_negotiation.cc:407] Connection header received
0928 05:22:52.838433 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.838436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.838501 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.838582 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.839777 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.840489 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.841649 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.842045 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.842134 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.845448 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.845452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.845455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.845485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.847835 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.848345 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.848348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.848350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.848473 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.848775 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.848778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.848780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.849006 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.849205 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.849360 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.849515 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:22:52.861882 9590 tablet.cc:2369] T 5fd4130b5caa4ab5ab2807f3ab801dcc P dba507306049476ea4bef69fa2227a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260928 05:22:52.881934 9456 tablet.cc:2369] T 5fd4130b5caa4ab5ab2807f3ab801dcc P d2e7d020166b469487feac488c238bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:22:52 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:52.889568 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.876557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.8.124.193:44449)
0928 05:22:52.876726 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.876732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.877034 (+ 302us) server_negotiation.cc:407] Connection header received
0928 05:22:52.877199 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.877202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.877283 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.877366 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.878716 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.879432 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.880589 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.881013 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.881124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.884836 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.884843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.884848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.884886 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.887558 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.888119 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.888122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.888125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.888247 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.888522 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.888525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.888526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.888763 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.888937 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.889120 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.889274 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 05:22:52.891202 9370 tablet_service.cc:2908] Rejecting scan request for tablet 5fd4130b5caa4ab5ab2807f3ab801dcc: Uninitialized: clean time has not yet been initialized
I20260928 05:22:52.893916 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9327
I20260928 05:22:52.904589 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:52.894684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.8.124.193:44449)
0928 05:22:52.894822 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:52.894825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:52.895184 (+ 359us) server_negotiation.cc:407] Connection header received
0928 05:22:52.895307 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:52.895311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:52.895360 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:52.895431 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:52.896437 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.897122 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.898241 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:52.898614 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:52.898698 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:52.900600 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:52.900604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:52.900607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:52.900628 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:52.902765 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.903289 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.903292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.903295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.903386 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:52.903627 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:52.903630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:52.903631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:52.903843 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:52.904000 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:52.904184 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:22:52.904360 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:22:52.905144 9450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:52.993500 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9327
I20260928 05:22:53.006729 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9461
Sep 28 05:22:53 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:22:53.021564 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:53.007626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.8.124.194:39753)
0928 05:22:53.007848 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:53.007854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:53.007874 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:22:53.007945 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:53.007950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:53.008011 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:53.008093 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:53.009294 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:53.010107 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:53.011424 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:53.011857 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:53.011967 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:53.015828 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:53.015843 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:53.015860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:53.015894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:53.018950 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:53.019585 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:53.019596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:53.019600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:53.019730 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:53.020071 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:53.020076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:53.020080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:53.020476 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:53.020757 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:53.021032 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 05:22:53.021241 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:22:53.022235 9584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:53.108153 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9461
I20260928 05:22:53.120935 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9594
Sep 28 05:22:53 dist-test-slave-515x krb5kdc[9221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:22:53.134629 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:53.121642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.8.124.195:36767)
0928 05:22:53.121791 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:53.121795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:53.121814 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:22:53.121958 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:53.121962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:53.122025 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:53.122109 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:53.123282 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:53.124063 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:53.125250 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:53.125752 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:53.125845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:53.129717 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:53.129732 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:53.129752 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:53.129783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:53.132304 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:53.132909 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:53.132915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:53.132917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:53.133043 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:53.133313 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:53.133316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:53.133318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:53.133679 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:53.133940 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:53.134170 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 05:22:53.134348 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:22:53.135243 9717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:53.223604 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9594
I20260928 05:22:53.236055 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9234
I20260928 05:22:53.237066 9296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:53.340945 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9234
2026-09-28T05:22:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3824 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:53 dist-test-slave-515x krb5kdc[9770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:53 dist-test-slave-515x krb5kdc[9770](info): set up 2 sockets
Sep 28 05:22:53 dist-test-slave-515x krb5kdc[9770](info): commencing operation
krb5kdc: starting...
W20260928 05:22:54.333421 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:22:54 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:22:54.490899 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36529
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:54.792753 9783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:54.793010 9783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:54.793098 9783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:54.799947 9783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:54.800034 9783 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:22:54.800079 9783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:54.800137 9783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:54.800190 9783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:54.800246 9783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:54.809628 9783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:54.810906 9783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:54.812157 9783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:54.822844 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:54.822806 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:54.822885 9789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:54.823707 9783 server_base.cc:1034] running on GCE node
I20260928 05:22:54.824366 9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:54.825627 9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:54.826848 9783 hybrid_clock.cc:648] HybridClock initialized: now 1790572974826786 us; error 42 us; skew 500 ppm
I20260928 05:22:54.827124 9783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:54 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572974, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:54.829967 9783 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:22:54.831265 9783 webserver.cc:460] Webserver started at http://127.8.124.254:39063/ using document root <none> and password file <none>
I20260928 05:22:54.831897 9783 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:54.831988 9783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:54.832276 9783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:54.835012 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "98df7dc3fd3d448ca8722cd517d26fab"
format_stamp: "Formatted at 2026-09-28 05:22:54 on dist-test-slave-515x"
server_key: "7b081e349af2e4791cfce8f3c2336caa"
server_key_iv: "c2514a20afdad78a44dcedbdcf8b4849"
server_key_version: "encryptionkey@0"
I20260928 05:22:54.835721 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "98df7dc3fd3d448ca8722cd517d26fab"
format_stamp: "Formatted at 2026-09-28 05:22:54 on dist-test-slave-515x"
server_key: "7b081e349af2e4791cfce8f3c2336caa"
server_key_iv: "c2514a20afdad78a44dcedbdcf8b4849"
server_key_version: "encryptionkey@0"
I20260928 05:22:54.840518 9783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:22:54.843721 9798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:54.845194 9783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:22:54.845362 9783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "98df7dc3fd3d448ca8722cd517d26fab"
format_stamp: "Formatted at 2026-09-28 05:22:54 on dist-test-slave-515x"
server_key: "7b081e349af2e4791cfce8f3c2336caa"
server_key_iv: "c2514a20afdad78a44dcedbdcf8b4849"
server_key_version: "encryptionkey@0"
I20260928 05:22:54.845544 9783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:54.858368 9783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:54.862277 9783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:54.862514 9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:54.880343 9783 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:36529
I20260928 05:22:54.880363 9859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:36529 every 8 connection(s)
I20260928 05:22:54.881532 9783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:54.884972 9860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:54.885807 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9783
I20260928 05:22:54.886015 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:22:54.886803 8691 external_mini_cluster.cc:1428] Setting key 5122341eb0d8ce5336d6c2d9e8194680
I20260928 05:22:54.898370 9860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Bootstrap starting.
Sep 28 05:22:54 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:54.901957 9860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:54.902938 9860 log.cc:824] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:54.905794 9860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: No bootstrap required, opened a new log
I20260928 05:22:54.908699 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:54.888746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.8.124.254:36529)
0928 05:22:54.889486 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:54.889498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:54.889539 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:22:54.891159 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:54.891180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:54.891581 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:54.891904 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:54.893538 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:54.894841 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:54.896511 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:54.897159 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:54.897277 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:54.901745 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:54.901752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:54.901765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:54.901800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:54.904780 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:54.905580 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:54.905586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:54.905591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:54.905816 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:54.906207 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:54.906211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:54.906213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:54.906691 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:54.906886 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:54.907288 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 05:22:54.907688 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":306,"threads_started":1}
I20260928 05:22:54.913249 9860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } }
I20260928 05:22:54.913552 9860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:54.913650 9860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7dc3fd3d448ca8722cd517d26fab, State: Initialized, Role: FOLLOWER
I20260928 05:22:54.914193 9860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } }
I20260928 05:22:54.914425 9860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:54.914517 9860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:54.914630 9860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:54.916814 9860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } }
I20260928 05:22:54.917238 9860 leader_election.cc:302] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98df7dc3fd3d448ca8722cd517d26fab; no voters:
I20260928 05:22:54.917774 9860 leader_election.cc:288] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:54.917870 9865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:54.918823 9865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [term 1 LEADER]: Becoming Leader. State: Replica: 98df7dc3fd3d448ca8722cd517d26fab, State: Running, Role: LEADER
I20260928 05:22:54.919268 9865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } }
I20260928 05:22:54.919669 9860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:54.921772 9866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98df7dc3fd3d448ca8722cd517d26fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } } }
I20260928 05:22:54.921818 9867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98df7dc3fd3d448ca8722cd517d26fab. Latest consensus state: current_term: 1 leader_uuid: "98df7dc3fd3d448ca8722cd517d26fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98df7dc3fd3d448ca8722cd517d26fab" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36529 } } }
I20260928 05:22:54.922154 9867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:54.922152 9866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:54.924052 9872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:54.928436 9872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:54.936298 9872 catalog_manager.cc:1347] Generated new cluster ID: 6e4bb8c3da2d4d699f7fd1cb92cef335
I20260928 05:22:54.936401 9872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:54.947250 9872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:54.948547 9872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:54.961035 9872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Generated new TSK 0
I20260928 05:22:54.961861 9872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:22:55.011171 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36529
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:55.323403 9886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:55.323664 9886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:55.323794 9886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:55.330873 9886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:55.330984 9886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:55.331094 9886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:22:55.331197 9886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:22:55.340812 9886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:55.342046 9886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:55.343348 9886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:55.355414 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:55.355446 9894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:55.355487 9892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:55.356555 9886 server_base.cc:1034] running on GCE node
I20260928 05:22:55.357111 9886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:55.357983 9886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:55.359177 9886 hybrid_clock.cc:648] HybridClock initialized: now 1790572975359112 us; error 41 us; skew 500 ppm
I20260928 05:22:55.359431 9886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:55 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:55.362308 9886 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:22:55.363557 9886 webserver.cc:460] Webserver started at http://127.8.124.193:39643/ using document root <none> and password file <none>
I20260928 05:22:55.364179 9886 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:55.364270 9886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:55.364552 9886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:55.367259 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "bebca9bd56804e06b0ff101a839f25fa"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "fe53408de9149dc8e74b6ddf2be6afbc"
server_key_iv: "7661a6fad29604f0d51d1f33d0dbaf91"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.367991 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "bebca9bd56804e06b0ff101a839f25fa"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "fe53408de9149dc8e74b6ddf2be6afbc"
server_key_iv: "7661a6fad29604f0d51d1f33d0dbaf91"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.374917 9886 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 05:22:55.378923 9901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:55.380265 9886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:55.380412 9886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "bebca9bd56804e06b0ff101a839f25fa"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "fe53408de9149dc8e74b6ddf2be6afbc"
server_key_iv: "7661a6fad29604f0d51d1f33d0dbaf91"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.380611 9886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:55.414399 9886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:55.418427 9886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:55.418676 9886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:55.421267 9886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:55.421344 9886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:55.421473 9886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:55.421528 9886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:22:55 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:55.441769 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.421790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:57855 (local address 127.8.124.254:36529)
0928 05:22:55.422393 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:55.422400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:55.423509 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 05:22:55.426313 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:55.426319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:55.426398 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:55.426494 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:55.428430 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.429131 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.430959 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.431624 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.431750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.435422 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:55.435429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:55.435434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:55.435473 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:55.438353 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.439385 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.439390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.439394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.439532 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.440071 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.440077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.440081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.440346 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:55.440447 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:55.441305 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 05:22:55.441453 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":383,"threads_started":1}
I20260928 05:22:55.443442 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.422519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.193:57855)
0928 05:22:55.423335 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:55.423389 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:55.425949 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:55.426756 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:55.426768 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:55.427481 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:55.428261 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.428286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.429375 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.429381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.430822 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.430833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.431817 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.431821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.431996 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.432591 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:55.432618 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:55.435174 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:55.438512 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.438521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.438538 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.439244 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.439638 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.439646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.439651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.439917 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.440468 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:55.440477 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:55.440908 (+ 431us) client_negotiation.cc:769] Sending connection context
0928 05:22:55.441196 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 05:22:55.441742 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
I20260928 05:22:55.462270 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.449356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:42213 (local address 127.8.124.254:36529)
0928 05:22:55.449492 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:55.449496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:55.450331 (+ 835us) server_negotiation.cc:407] Connection header received
0928 05:22:55.450497 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:55.450503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:55.450564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:55.450637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:55.451470 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.452151 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.453365 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.453830 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.453932 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.456280 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:55.456285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:55.456288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:55.456315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:55.459215 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.460084 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.460090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.460093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.460240 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.460678 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.460682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.460685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.460964 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:55.461103 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:55.461844 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 05:22:55.462006 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:22:55.462577 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.449453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.193:42213)
0928 05:22:55.450245 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:55.450261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:55.450382 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:55.450670 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:55.450674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:55.451079 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:55.451339 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.451347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.452272 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.452278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.453244 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.453254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.454018 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.454024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.454197 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.454915 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:55.454940 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:55.456108 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:55.459354 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.459360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.459363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.459920 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.460339 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.460344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.460348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.460524 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.461265 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:55.461271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:55.461546 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 05:22:55.461747 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 05:22:55.462101 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":436,"threads_started":1}
I20260928 05:22:55.472754 9886 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:43629
I20260928 05:22:55.472772 10021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:43629 every 8 connection(s)
I20260928 05:22:55.473935 9886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:22:55.475736 10022 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36529
I20260928 05:22:55.475926 10022 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:55.476493 10022 heartbeater.cc:511] Master 127.8.124.254:36529 requested a full tablet report, sending...
I20260928 05:22:55.478291 9815 ts_manager.cc:194] Registered new tserver with Master: bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629)
I20260928 05:22:55.479096 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 9886
I20260928 05:22:55.479280 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:22:55.479696 9815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:57855
I20260928 05:22:55.479893 8691 external_mini_cluster.cc:1428] Setting key d4796aa7c33eb7e2cd6147f501cc8596
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:22:55.514087 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36529
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:55.815357 10026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:55.815624 10026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:55.815732 10026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:55.822979 10026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:55.823089 10026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:55.823194 10026 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:22:55.823299 10026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:22:55.833117 10026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:55.834385 10026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:55.835675 10026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:55.848049 10032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:55.848078 10031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:55.848130 10034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:55.849016 10026 server_base.cc:1034] running on GCE node
I20260928 05:22:55.849588 10026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:55.850533 10026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:55.851748 10026 hybrid_clock.cc:648] HybridClock initialized: now 1790572975851673 us; error 50 us; skew 500 ppm
I20260928 05:22:55.852036 10026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:55 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:55.854996 10026 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:22:55.856405 10026 webserver.cc:460] Webserver started at http://127.8.124.194:44617/ using document root <none> and password file <none>
I20260928 05:22:55.857206 10026 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:55.857298 10026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:55.857581 10026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:55.860296 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "285cc9bcb7f948479e7d63b976900d21"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "40d9024bcdfdfe462e9fdd870b40635e"
server_key_iv: "a4320ef37550086c43f8b77af711454b"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.860998 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "285cc9bcb7f948479e7d63b976900d21"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "40d9024bcdfdfe462e9fdd870b40635e"
server_key_iv: "a4320ef37550086c43f8b77af711454b"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.865576 10026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 05:22:55.868827 10041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:55.870178 10026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:22:55.870316 10026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "285cc9bcb7f948479e7d63b976900d21"
format_stamp: "Formatted at 2026-09-28 05:22:55 on dist-test-slave-515x"
server_key: "40d9024bcdfdfe462e9fdd870b40635e"
server_key_iv: "a4320ef37550086c43f8b77af711454b"
server_key_version: "encryptionkey@0"
I20260928 05:22:55.870496 10026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:55.907058 10026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:55.911391 10026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:55.911698 10026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:55.914477 10026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:55.914582 10026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:55.914726 10026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:55.914803 10026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:22:55 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:55.937150 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.914994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57349 (local address 127.8.124.254:36529)
0928 05:22:55.915205 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:55.915211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:55.916667 (+ 1456us) server_negotiation.cc:407] Connection header received
0928 05:22:55.918849 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:55.918855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:55.918941 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:55.919026 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:55.921220 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.922126 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.924982 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.925583 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.925726 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.930988 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:55.930995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:55.931000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:55.931042 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:55.933999 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.934877 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.934883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.934887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.935012 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.935438 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.935443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.935447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.935739 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:55.935868 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:55.936607 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 05:22:55.936823 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:22:55.938247 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.915725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.194:57349)
0928 05:22:55.916503 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:55.916548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:55.918574 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:55.919372 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:55.919384 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:55.920152 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:55.921041 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.921062 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.922326 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.922332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.924796 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.924823 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.926041 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.926049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.926416 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.927195 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:55.927232 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:55.930695 (+ 3463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:55.934131 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.934144 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.934165 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.934765 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.935120 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.935125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.935129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.935335 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.935876 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:55.935884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:55.936247 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 05:22:55.936502 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 05:22:55.937021 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":332,"threads_started":1}
I20260928 05:22:55.963899 10026 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:43717
I20260928 05:22:55.964075 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:43717 every 8 connection(s)
I20260928 05:22:55.965191 10026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:22:55.967257 10161 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36529
I20260928 05:22:55.967478 10161 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:55.968370 10161 heartbeater.cc:511] Master 127.8.124.254:36529 requested a full tablet report, sending...
I20260928 05:22:55.969066 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10026
I20260928 05:22:55.969259 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:22:55.969802 8691 external_mini_cluster.cc:1428] Setting key 6af32861e7d7d46c04b5f7ad216a4974
I20260928 05:22:55.969873 9815 ts_manager.cc:194] Registered new tserver with Master: 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717)
I20260928 05:22:55.971159 9815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:57349
I20260928 05:22:55.979457 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.960649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:36265 (local address 127.8.124.254:36529)
0928 05:22:55.960862 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:55.960869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:55.967285 (+ 6416us) server_negotiation.cc:407] Connection header received
0928 05:22:55.967354 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:55.967358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:55.967437 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:55.967533 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:55.968677 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.969741 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.971520 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.972002 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.972123 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.974455 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:55.974460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:55.974463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:55.974492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:55.977106 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.977707 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.977713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.977717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.977849 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:55.978171 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:55.978175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:55.978178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:55.978467 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:55.978603 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:55.978957 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:22:55.979139 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:22:55.979533 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:55.960550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.194:36265)
0928 05:22:55.967094 (+ 6544us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:55.967113 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:55.967253 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:55.967619 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:55.967624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:55.968199 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:55.968530 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.968543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.969892 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.969897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.971337 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:55.971351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:55.972195 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:55.972202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:55.972407 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:55.973103 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:55.973125 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:55.974295 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:55.977218 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.977224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.977228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.977586 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.977945 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:55.977950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:55.977954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:55.978071 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:55.978573 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:55.978578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:55.978738 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 05:22:55.978846 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:22:55.979116 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6407,"thread_start_us":6778,"threads_started":1}
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:22:56.004865 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36529
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:56.311805 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:56.312076 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:56.312194 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:56.319069 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:56.319176 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:56.319314 10165 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:22:56.319422 10165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:22:56.328853 10165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:56.330121 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:56.331441 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:56.343533 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:56.343556 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:56.344049 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:56.344156 10165 server_base.cc:1034] running on GCE node
I20260928 05:22:56.344745 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:56.345683 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:56.346895 10165 hybrid_clock.cc:648] HybridClock initialized: now 1790572976346838 us; error 33 us; skew 500 ppm
I20260928 05:22:56.347164 10165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:56 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:56.350188 10165 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:22:56.351543 10165 webserver.cc:460] Webserver started at http://127.8.124.195:41945/ using document root <none> and password file <none>
I20260928 05:22:56.352212 10165 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:56.352294 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:56.352578 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:56.355280 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "7f421b0b3c5d474597a7fe044e4181b8"
format_stamp: "Formatted at 2026-09-28 05:22:56 on dist-test-slave-515x"
server_key: "ccac6c8142e7f4bcd2075767945df092"
server_key_iv: "e4d48809f290adc806ab975cd88d56e6"
server_key_version: "encryptionkey@0"
I20260928 05:22:56.355999 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "7f421b0b3c5d474597a7fe044e4181b8"
format_stamp: "Formatted at 2026-09-28 05:22:56 on dist-test-slave-515x"
server_key: "ccac6c8142e7f4bcd2075767945df092"
server_key_iv: "e4d48809f290adc806ab975cd88d56e6"
server_key_version: "encryptionkey@0"
I20260928 05:22:56.360762 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:22:56.364034 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:56.365322 10165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:56.365453 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "7f421b0b3c5d474597a7fe044e4181b8"
format_stamp: "Formatted at 2026-09-28 05:22:56 on dist-test-slave-515x"
server_key: "ccac6c8142e7f4bcd2075767945df092"
server_key_iv: "e4d48809f290adc806ab975cd88d56e6"
server_key_version: "encryptionkey@0"
I20260928 05:22:56.365675 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:56.387512 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:56.391590 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:56.391839 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:56.394343 10165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:56.394425 10165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:56.394534 10165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:56.394595 10165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:22:56 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:56.416990 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.394870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:44619 (local address 127.8.124.254:36529)
0928 05:22:56.395061 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.395067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.396488 (+ 1421us) server_negotiation.cc:407] Connection header received
0928 05:22:56.398529 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.398534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.398620 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.398719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:56.401080 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.402065 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.404357 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.404892 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.405000 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.409512 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:56.409519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:56.409524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:56.409616 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:56.413041 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.414073 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.414078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.414080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.414231 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.414860 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.414864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.414868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.415172 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:56.415329 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.416433 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.416649 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:22:56.418226 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.395491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.195:44619)
0928 05:22:56.396314 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.396359 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.398258 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.399015 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.399028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.399866 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:56.400866 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.400895 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.402226 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.402230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.404162 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.404178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.405115 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.405123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.405327 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.406007 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:56.406035 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:56.409235 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:56.413201 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.413212 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.413232 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.413885 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.414368 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.414374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.414379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.414698 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.415437 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:56.415447 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:56.415998 (+ 551us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.416326 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.416863 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":335,"threads_started":1}
I20260928 05:22:56.436208 10165 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:42165
I20260928 05:22:56.436208 10299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:42165 every 8 connection(s)
I20260928 05:22:56.436815 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.423372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:37503 (local address 127.8.124.254:36529)
0928 05:22:56.423516 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.423519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.424132 (+ 613us) server_negotiation.cc:407] Connection header received
0928 05:22:56.424369 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.424375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.424467 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.424570 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:56.425774 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.426683 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.428357 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.428853 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.428964 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.431289 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:56.431293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:56.431296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:56.431323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:56.434229 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.434947 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.434951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.434953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.435063 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.435464 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.435469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.435472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.435734 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:56.435872 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.436254 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.436443 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:22:56.436885 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.423478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.8.124.195:37503)
0928 05:22:56.424041 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.424060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.424174 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.424586 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.424589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.425053 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:56.425640 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.425654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.426869 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.426874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.428232 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.428244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.429171 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.429177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.429382 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.429987 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:56.430011 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:56.431065 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:56.434387 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.434392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.434396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.434851 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.435174 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.435180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.435184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.435317 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.435852 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:56.435856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:56.436029 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.436153 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.436466 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":315,"threads_started":1}
I20260928 05:22:56.437572 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:22:56.439793 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10165
I20260928 05:22:56.439992 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:22:56.440388 8691 external_mini_cluster.cc:1428] Setting key e68646ab68cdde96f82d7d4dbe77dab8
I20260928 05:22:56.439555 10300 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36529
I20260928 05:22:56.441036 10300 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:56.441663 10300 heartbeater.cc:511] Master 127.8.124.254:36529 requested a full tablet report, sending...
I20260928 05:22:56.442862 9814 ts_manager.cc:194] Registered new tserver with Master: 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165)
I20260928 05:22:56.443769 9814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:44619
I20260928 05:22:56.454924 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:22:56 dist-test-slave-515x krb5kdc[9770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260928 05:22:56.482575 10022 heartbeater.cc:503] Master 127.8.124.254:36529 was elected leader, sending a full tablet report...
Sep 28 05:22:56 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:56.497107 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.484568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.8.124.254:36529)
0928 05:22:56.484762 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.484770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.485045 (+ 275us) server_negotiation.cc:407] Connection header received
0928 05:22:56.485202 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.485206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.485284 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.485364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:56.486645 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.487359 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.488503 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.488897 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.488985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.492560 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:56.492565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:56.492569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:56.492601 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:56.494981 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.495561 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.495564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.495566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.495671 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.495964 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.495967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.495968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.496213 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:56.496427 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.496638 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.496795 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:22:56.500929 9814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:22:56.507313 9814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:22:56.530844 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.519209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.8.124.195:42165)
0928 05:22:56.520278 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.520287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.521314 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 05:22:56.521718 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.521727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.521970 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.522128 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.523151 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.524844 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.528204 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.529671 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.529824 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.530054 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.530340 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.530518 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":693,"threads_started":1}
I20260928 05:22:56.531637 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.519578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.8.124.193:43629)
0928 05:22:56.520660 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.520669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.521313 (+ 644us) server_negotiation.cc:407] Connection header received
0928 05:22:56.521732 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.521738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.521970 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.522131 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.523914 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.525291 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.528740 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.530200 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.530327 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.530540 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.531193 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.531363 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":845,"threads_started":1}
I20260928 05:22:56.531946 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.518915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42165 (local address 127.0.0.1:39348)
0928 05:22:56.521154 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.521200 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.521471 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.522467 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.522473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.522502 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.522926 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.522938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.525046 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.525051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.527407 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.527422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.529849 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.529855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.530053 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.530100 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.531353 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.531558 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1954,"mutex_wait_us":3,"spinlock_wait_cycles":3584,"thread_start_us":2001,"threads_started":1}
I20260928 05:22:56.532418 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.519049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:43629 (local address 127.0.0.1:51876)
0928 05:22:56.521161 (+ 2112us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.521200 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.521470 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.523341 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.523345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.523368 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.523741 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.523753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.525475 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.525479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.528566 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.528580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.530787 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.530796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.530987 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.531021 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.531879 (+ 858us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.532018 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"mutex_wait_us":234,"spinlock_wait_cycles":5632,"thread_start_us":1670,"threads_started":1}
I20260928 05:22:56.535008 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.521593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43717 (local address 127.0.0.1:58296)
0928 05:22:56.527977 (+ 6384us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.527992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.528086 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.528637 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.528642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.528665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.528991 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.528999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.530539 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.530544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.532826 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.532839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.534314 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.534321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.534486 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.534517 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.534605 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.534725 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6249,"thread_start_us":6106,"threads_started":1}
I20260928 05:22:56.535277 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.521861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.8.124.194:43717)
0928 05:22:56.527667 (+ 5806us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.527676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.528081 (+ 405us) server_negotiation.cc:407] Connection header received
0928 05:22:56.528255 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.528262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.528458 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.528622 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.529149 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.530389 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.532979 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.534111 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.534252 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.534450 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.534680 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.534848 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5645,"thread_start_us":5522,"threads_started":1}
I20260928 05:22:56.535902 10230 tablet_service.cc:1467] Processing CreateTablet for tablet 035f6afac0394f2f9f431a700fded3fe (DEFAULT_TABLE table=test-table [id=977e0630b6e34636b35edb856cbb0b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.536082 9952 tablet_service.cc:1467] Processing CreateTablet for tablet 035f6afac0394f2f9f431a700fded3fe (DEFAULT_TABLE table=test-table [id=977e0630b6e34636b35edb856cbb0b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.537285 10230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035f6afac0394f2f9f431a700fded3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.537415 9952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035f6afac0394f2f9f431a700fded3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.538687 10091 tablet_service.cc:1467] Processing CreateTablet for tablet 035f6afac0394f2f9f431a700fded3fe (DEFAULT_TABLE table=test-table [id=977e0630b6e34636b35edb856cbb0b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.540055 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035f6afac0394f2f9f431a700fded3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.549652 10319 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Bootstrap starting.
I20260928 05:22:56.552488 10319 tablet_bootstrap.cc:654] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.553522 10319 log.cc:824] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:56.553885 10321 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Bootstrap starting.
I20260928 05:22:56.556777 10320 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Bootstrap starting.
I20260928 05:22:56.558557 10321 tablet_bootstrap.cc:654] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.559688 10320 tablet_bootstrap.cc:654] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.559722 10321 log.cc:824] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:56.560513 10319 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: No bootstrap required, opened a new log
I20260928 05:22:56.560750 10319 ts_tablet_manager.cc:1397] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260928 05:22:56.560714 10320 log.cc:824] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:56.562531 10321 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: No bootstrap required, opened a new log
I20260928 05:22:56.562805 10321 ts_tablet_manager.cc:1397] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:22:56.563488 10320 tablet_bootstrap.cc:492] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: No bootstrap required, opened a new log
I20260928 05:22:56.563727 10320 ts_tablet_manager.cc:1397] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:22:56.568312 10319 raft_consensus.cc:348] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.568792 10319 raft_consensus.cc:374] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.568892 10319 raft_consensus.cc:729] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f421b0b3c5d474597a7fe044e4181b8, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.569636 10319 consensus_queue.cc:260] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.570791 10300 heartbeater.cc:503] Master 127.8.124.254:36529 was elected leader, sending a full tablet report...
I20260928 05:22:56.571477 10319 ts_tablet_manager.cc:1428] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260928 05:22:56.572865 10321 raft_consensus.cc:348] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.573199 10321 raft_consensus.cc:374] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.573287 10321 raft_consensus.cc:729] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285cc9bcb7f948479e7d63b976900d21, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.573894 10321 consensus_queue.cc:260] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.574488 10320 raft_consensus.cc:348] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.574841 10161 heartbeater.cc:503] Master 127.8.124.254:36529 was elected leader, sending a full tablet report...
I20260928 05:22:56.574967 10320 raft_consensus.cc:374] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.575060 10320 raft_consensus.cc:729] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebca9bd56804e06b0ff101a839f25fa, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.575599 10321 ts_tablet_manager.cc:1428] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 05:22:56.575546 10320 consensus_queue.cc:260] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.577054 10320 ts_tablet_manager.cc:1428] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 05:22:56.626109 10327 raft_consensus.cc:482] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:56.626385 10327 raft_consensus.cc:504] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.627944 10327 leader_election.cc:288] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165), 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717)
I20260928 05:22:56.633864 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.628343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:57189 (local address 127.8.124.195:42165)
0928 05:22:56.628515 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.628522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.628945 (+ 423us) server_negotiation.cc:407] Connection header received
0928 05:22:56.629139 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.629146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.629235 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.629331 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.629919 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.630610 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.632288 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.633012 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.633150 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.633202 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.633509 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.633623 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:22:56.633920 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.628282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42165 (local address 127.8.124.193:57189)
0928 05:22:56.628828 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.628842 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.628987 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.629413 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.629418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.629442 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.629792 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.629801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.630730 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.630735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.632134 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.632145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.633195 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.633200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.633357 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.633391 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.633485 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.633621 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":294,"threads_started":1}
I20260928 05:22:56.634450 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.629223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:60039 (local address 127.8.124.194:43717)
0928 05:22:56.629396 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.629401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.629581 (+ 180us) server_negotiation.cc:407] Connection header received
0928 05:22:56.629770 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.629782 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.629837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.629911 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.630369 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.631064 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.632607 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.633523 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.633677 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.633741 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.634101 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.634228 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:22:56.634449 10325 raft_consensus.cc:482] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:56.634958 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.628980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43717 (local address 127.8.124.193:60039)
0928 05:22:56.629498 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.629511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.629651 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.629932 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.629935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.629956 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.630240 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.630248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.631186 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.631190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.632472 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.632482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.633753 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.633760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.633922 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.633951 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.634613 (+ 662us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.634732 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":228,"threads_started":1}
I20260928 05:22:56.635032 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "bebca9bd56804e06b0ff101a839f25fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8" is_pre_election: true
I20260928 05:22:56.634840 10325 raft_consensus.cc:504] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.635809 10113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "bebca9bd56804e06b0ff101a839f25fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285cc9bcb7f948479e7d63b976900d21" is_pre_election: true
I20260928 05:22:56.635885 10250 raft_consensus.cc:2454] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebca9bd56804e06b0ff101a839f25fa in term 0.
I20260928 05:22:56.636207 10113 raft_consensus.cc:2454] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebca9bd56804e06b0ff101a839f25fa in term 0.
I20260928 05:22:56.636551 9904 leader_election.cc:302] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f421b0b3c5d474597a7fe044e4181b8, bebca9bd56804e06b0ff101a839f25fa; no voters:
I20260928 05:22:56.636943 10327 raft_consensus.cc:2781] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:22:56.637090 10327 raft_consensus.cc:482] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:22:56.637194 10327 raft_consensus.cc:3037] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.638620 10325 leader_election.cc:288] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629), 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717)
I20260928 05:22:56.640293 10327 raft_consensus.cc:504] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.641265 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "bebca9bd56804e06b0ff101a839f25fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8"
I20260928 05:22:56.641461 10250 raft_consensus.cc:3037] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.641542 10113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "bebca9bd56804e06b0ff101a839f25fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285cc9bcb7f948479e7d63b976900d21"
I20260928 05:22:56.641794 10113 raft_consensus.cc:3037] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.644688 10250 raft_consensus.cc:2454] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebca9bd56804e06b0ff101a839f25fa in term 1.
I20260928 05:22:56.644956 10113 raft_consensus.cc:2454] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebca9bd56804e06b0ff101a839f25fa in term 1.
I20260928 05:22:56.645205 9904 leader_election.cc:302] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f421b0b3c5d474597a7fe044e4181b8, bebca9bd56804e06b0ff101a839f25fa; no voters:
I20260928 05:22:56.645305 10327 leader_election.cc:288] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [CANDIDATE]: Term 1 election: Requested vote from peers 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165), 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717)
I20260928 05:22:56.645500 10327 raft_consensus.cc:2781] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:56.645521 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.637417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43717 (local address 127.8.124.195:47647)
0928 05:22:56.637980 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.637994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.638090 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.638816 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.638820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.638842 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.639073 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.639080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.640085 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.640090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.642274 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.642289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.644405 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.644410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.644608 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.644636 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.644983 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.645106 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":26,"thread_start_us":957,"threads_started":1}
I20260928 05:22:56.645646 10327 raft_consensus.cc:686] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 1 LEADER]: Becoming Leader. State: Replica: bebca9bd56804e06b0ff101a839f25fa, State: Running, Role: LEADER
I20260928 05:22:56.645572 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.637217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:43629 (local address 127.8.124.195:33893)
0928 05:22:56.637400 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.637419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.637516 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.638311 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.638315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.638336 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.638515 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.638525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.639800 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.639805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.642869 (+ 3064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.642882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.644481 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.644487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.644675 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.644709 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.644796 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.644921 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1041}
I20260928 05:22:56.646127 10327 consensus_queue.cc:237] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } }
I20260928 05:22:56.647930 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.637851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:33893 (local address 127.8.124.193:43629)
0928 05:22:56.638038 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.638044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.638061 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:56.638113 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.638118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.638200 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.638291 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.638648 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.639675 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.643049 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.644192 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.644338 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.644400 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.645067 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.645209 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:56.648629 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.637964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47647 (local address 127.8.124.194:43717)
0928 05:22:56.638101 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.638106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.638121 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:22:56.638169 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.638173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.638239 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.638308 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.639193 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.639941 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.642455 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.644044 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.644164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.644231 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.644793 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.644919 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:22:56.648802 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "7f421b0b3c5d474597a7fe044e4181b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebca9bd56804e06b0ff101a839f25fa" is_pre_election: true
I20260928 05:22:56.649102 10113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035f6afac0394f2f9f431a700fded3fe" candidate_uuid: "7f421b0b3c5d474597a7fe044e4181b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285cc9bcb7f948479e7d63b976900d21" is_pre_election: true
I20260928 05:22:56.649384 10113 raft_consensus.cc:2379] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f421b0b3c5d474597a7fe044e4181b8 in current term 1: Already voted for candidate bebca9bd56804e06b0ff101a839f25fa in this term.
I20260928 05:22:56.650683 10181 leader_election.cc:302] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f421b0b3c5d474597a7fe044e4181b8; no voters: 285cc9bcb7f948479e7d63b976900d21, bebca9bd56804e06b0ff101a839f25fa
I20260928 05:22:56.651104 10325 raft_consensus.cc:2726] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:22:56.652240 9815 catalog_manager.cc:5495] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa reported cstate change: term changed from 0 to 1, leader changed from <none> to bebca9bd56804e06b0ff101a839f25fa (127.8.124.193). New cstate: current_term: 1 leader_uuid: "bebca9bd56804e06b0ff101a839f25fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } health_report { overall_health: UNKNOWN } } }
I20260928 05:22:56.688452 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.683467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.8.124.193:43629)
0928 05:22:56.683612 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.683616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.683709 (+ 93us) server_negotiation.cc:407] Connection header received
0928 05:22:56.683962 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.683965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.684048 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.684173 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:56.684496 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.685275 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.686554 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.686929 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.687016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.687407 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:56.687920 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:56.688047 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.688139 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.688223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 05:22:56.689136 10301 tablet.cc:2369] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:22:56.696173 10113 raft_consensus.cc:1261] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Refusing update from remote peer bebca9bd56804e06b0ff101a839f25fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:56.696230 10250 raft_consensus.cc:1261] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Refusing update from remote peer bebca9bd56804e06b0ff101a839f25fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:56.696973 10331 consensus_queue.cc:1035] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:56.697324 10327 consensus_queue.cc:1035] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:56.707010 10334 mvcc.cc:205] Tried to move back new op lower bound from 7334186912538419200 to 7334186912345755648. Current Snapshot: MvccSnapshot[applied={T|T < 7334186912538419200}]
I20260928 05:22:56.708848 10336 mvcc.cc:205] Tried to move back new op lower bound from 7334186912538419200 to 7334186912345755648. Current Snapshot: MvccSnapshot[applied={T|T < 7334186912538419200}]
I20260928 05:22:56.724782 9815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:56.725077 9815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:56.728662 9815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Sending DeleteTablet for 3 replicas of tablet 035f6afac0394f2f9f431a700fded3fe
I20260928 05:22:56.729895 10230 tablet_service.cc:1514] Processing DeleteTablet for tablet 035f6afac0394f2f9f431a700fded3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:56 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39348
I20260928 05:22:56.729871 9952 tablet_service.cc:1514] Processing DeleteTablet for tablet 035f6afac0394f2f9f431a700fded3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:56 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:51876
I20260928 05:22:56.729890 10091 tablet_service.cc:1514] Processing DeleteTablet for tablet 035f6afac0394f2f9f431a700fded3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:56 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:58296
I20260928 05:22:56.732391 9815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:22:56.733876 9815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:22:56.744240 10090 tablet_service.cc:1467] Processing CreateTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 (DEFAULT_TABLE table=test-table [id=355bd4225a6445219744da2bd4c9670d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.744807 10090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57407bd9fc344bc1b96a71452dbf65c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.744879 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 (DEFAULT_TABLE table=test-table [id=355bd4225a6445219744da2bd4c9670d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.745416 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57407bd9fc344bc1b96a71452dbf65c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.750872 10321 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Bootstrap starting.
I20260928 05:22:56.751331 10229 tablet_service.cc:1467] Processing CreateTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 (DEFAULT_TABLE table=test-table [id=355bd4225a6445219744da2bd4c9670d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:22:56.751878 10229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57407bd9fc344bc1b96a71452dbf65c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.752404 10321 tablet_bootstrap.cc:654] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.754176 10344 tablet_replica.cc:330] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: stopping tablet replica
I20260928 05:22:56.754807 10344 raft_consensus.cc:2229] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:22:56.755196 10320 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Bootstrap starting.
I20260928 05:22:56.756927 10320 tablet_bootstrap.cc:654] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.758551 10319 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Bootstrap starting.
I20260928 05:22:56.760172 10319 tablet_bootstrap.cc:654] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.764540 10344 raft_consensus.cc:2258] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:56.766259 10321 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: No bootstrap required, opened a new log
I20260928 05:22:56.766426 10321 ts_tablet_manager.cc:1397] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260928 05:22:56.767206 10321 raft_consensus.cc:348] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.767448 10321 raft_consensus.cc:374] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.767539 10321 raft_consensus.cc:729] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285cc9bcb7f948479e7d63b976900d21, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.767830 10321 consensus_queue.cc:260] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.770598 10344 ts_tablet_manager.cc:1905] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:56.771160 10321 ts_tablet_manager.cc:1428] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:22:56.775781 10346 tablet_replica.cc:330] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: stopping tablet replica
I20260928 05:22:56.776221 10346 raft_consensus.cc:2229] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:56.777385 10346 raft_consensus.cc:2258] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:56.778923 10320 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: No bootstrap required, opened a new log
I20260928 05:22:56.779152 10320 ts_tablet_manager.cc:1397] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20260928 05:22:56.779438 10346 ts_tablet_manager.cc:1905] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:56.780184 10320 raft_consensus.cc:348] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.780530 10320 raft_consensus.cc:374] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.780663 10320 raft_consensus.cc:729] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebca9bd56804e06b0ff101a839f25fa, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.781008 10320 consensus_queue.cc:260] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.781665 10320 ts_tablet_manager.cc:1428] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 05:22:56.781926 10345 tablet_replica.cc:330] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: stopping tablet replica
I20260928 05:22:56.782418 10345 raft_consensus.cc:2229] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:56.782953 10345 raft_consensus.cc:2258] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:56.786562 10346 ts_tablet_manager.cc:1918] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:56.786727 10346 log.cc:1196] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/035f6afac0394f2f9f431a700fded3fe
I20260928 05:22:56.786725 10345 ts_tablet_manager.cc:1905] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:56.787225 10346 ts_tablet_manager.cc:1939] T 035f6afac0394f2f9f431a700fded3fe P 285cc9bcb7f948479e7d63b976900d21: Deleting consensus metadata
I20260928 05:22:56.788962 9801 catalog_manager.cc:4841] TS 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717): tablet 035f6afac0394f2f9f431a700fded3fe (table test-table [id=977e0630b6e34636b35edb856cbb0b59]) successfully deleted
I20260928 05:22:56.786854 10319 tablet_bootstrap.cc:492] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: No bootstrap required, opened a new log
I20260928 05:22:56.789201 10344 ts_tablet_manager.cc:1918] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:56.789362 10344 log.cc:1196] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/035f6afac0394f2f9f431a700fded3fe
I20260928 05:22:56.789417 10319 ts_tablet_manager.cc:1397] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.003s
I20260928 05:22:56.789928 10344 ts_tablet_manager.cc:1939] T 035f6afac0394f2f9f431a700fded3fe P bebca9bd56804e06b0ff101a839f25fa: Deleting consensus metadata
I20260928 05:22:56.790252 10319 raft_consensus.cc:348] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.790468 10319 raft_consensus.cc:374] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.790566 10319 raft_consensus.cc:729] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f421b0b3c5d474597a7fe044e4181b8, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.790898 10319 consensus_queue.cc:260] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.791517 9799 catalog_manager.cc:4841] TS bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629): tablet 035f6afac0394f2f9f431a700fded3fe (table test-table [id=977e0630b6e34636b35edb856cbb0b59]) successfully deleted
I20260928 05:22:56.792042 10319 ts_tablet_manager.cc:1428] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:56.794188 10326 raft_consensus.cc:482] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:56.794358 10326 raft_consensus.cc:504] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.794910 10345 ts_tablet_manager.cc:1918] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:22:56.795048 10345 log.cc:1196] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/035f6afac0394f2f9f431a700fded3fe
I20260928 05:22:56.795528 10345 ts_tablet_manager.cc:1939] T 035f6afac0394f2f9f431a700fded3fe P 7f421b0b3c5d474597a7fe044e4181b8: Deleting consensus metadata
I20260928 05:22:56.795915 10326 leader_election.cc:288] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629), 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165)
I20260928 05:22:56.797043 9801 catalog_manager.cc:4841] TS 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165): tablet 035f6afac0394f2f9f431a700fded3fe (table test-table [id=977e0630b6e34636b35edb856cbb0b59]) successfully deleted
I20260928 05:22:56.804020 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.796118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:43629 (local address 127.8.124.194:54231)
0928 05:22:56.798256 (+ 2138us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.798275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.798403 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.798892 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.798896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.798917 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.799223 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.799236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.800531 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.800537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.802112 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.802123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.803271 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.803278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.803452 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.803486 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.803576 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.803700 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"thread_start_us":2484,"threads_started":1}
I20260928 05:22:56.804037 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.796458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:54231 (local address 127.8.124.193:43629)
0928 05:22:56.796621 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.796626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.798340 (+ 1714us) server_negotiation.cc:407] Connection header received
0928 05:22:56.798589 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.798595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.798669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.798752 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.799395 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.800367 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.802257 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.803083 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.803233 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.803307 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.803611 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.803759 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:22:56.804863 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57407bd9fc344bc1b96a71452dbf65c9" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebca9bd56804e06b0ff101a839f25fa" is_pre_election: true
I20260928 05:22:56.805119 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.797320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:55111 (local address 127.8.124.195:42165)
0928 05:22:56.797455 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.797460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.798960 (+ 1500us) server_negotiation.cc:407] Connection header received
0928 05:22:56.799115 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.799120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.799184 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.799271 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:22:56.799814 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.800774 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.803164 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.804191 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.804340 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.804401 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.804769 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.804896 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 05:22:56.805155 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.796676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42165 (local address 127.8.124.194:55111)
0928 05:22:56.798860 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.798877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.799002 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.799301 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.799305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.799329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:22:56.799655 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.799666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.800905 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.800910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.803010 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.803024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.804523 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.804527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.804635 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.804661 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.804747 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.804856 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"thread_start_us":1543,"threads_started":1}
I20260928 05:22:56.805153 9977 raft_consensus.cc:2454] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 0.
I20260928 05:22:56.805706 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57407bd9fc344bc1b96a71452dbf65c9" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8" is_pre_election: true
I20260928 05:22:56.805883 10250 raft_consensus.cc:2454] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 0.
I20260928 05:22:56.805891 10042 leader_election.cc:302] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285cc9bcb7f948479e7d63b976900d21, bebca9bd56804e06b0ff101a839f25fa; no voters:
I20260928 05:22:56.806463 10326 raft_consensus.cc:2781] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:22:56.806562 10326 raft_consensus.cc:482] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:22:56.806643 10326 raft_consensus.cc:3037] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.808601 10326 raft_consensus.cc:504] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.809123 10326 leader_election.cc:288] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 election: Requested vote from peers bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629), 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165)
I20260928 05:22:56.809552 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57407bd9fc344bc1b96a71452dbf65c9" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8"
I20260928 05:22:56.809551 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57407bd9fc344bc1b96a71452dbf65c9" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebca9bd56804e06b0ff101a839f25fa"
I20260928 05:22:56.809757 10250 raft_consensus.cc:3037] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.809799 9977 raft_consensus.cc:3037] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:56.811744 10250 raft_consensus.cc:2454] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 1.
I20260928 05:22:56.812170 10044 leader_election.cc:302] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285cc9bcb7f948479e7d63b976900d21, 7f421b0b3c5d474597a7fe044e4181b8; no voters:
I20260928 05:22:56.812758 10326 raft_consensus.cc:2781] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:56.813026 9977 raft_consensus.cc:2454] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 1.
I20260928 05:22:56.813469 10326 raft_consensus.cc:686] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 1 LEADER]: Becoming Leader. State: Replica: 285cc9bcb7f948479e7d63b976900d21, State: Running, Role: LEADER
I20260928 05:22:56.813885 10326 consensus_queue.cc:237] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.817143 9815 catalog_manager.cc:5495] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:22:56 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572974, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:56.873914 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.860013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36529 (local address 127.0.0.1:60160)
0928 05:22:56.860618 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:56.860638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:56.860788 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:56.861107 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:56.861112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:56.862017 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:56.862376 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.862387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.863370 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.863374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.864334 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:56.864345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.865025 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.865031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:56.865163 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.865785 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:56.865803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:56.867800 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:56.871424 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.871433 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.871438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.871837 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.872207 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:56.872210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:56.872213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:56.872333 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:56.872922 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:56.872928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:56.873076 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 05:22:56.873279 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 05:22:56.873507 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":328,"threads_started":1}
I20260928 05:22:56.874105 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:56.860004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.8.124.254:36529)
0928 05:22:56.860203 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:56.860209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:56.860724 (+ 515us) server_negotiation.cc:407] Connection header received
0928 05:22:56.860892 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:56.860896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:56.860968 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:56.861048 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:56.862535 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.863245 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.864464 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:56.864861 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:56.864963 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:56.867958 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:56.867964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:56.867969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:56.868005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:56.871299 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.871942 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.871947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.871951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.872103 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:56.872424 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:56.872429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:56.872433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:56.872801 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:56.872941 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:56.873362 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 05:22:56.873737 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":61}
I20260928 05:22:56.879228 9815 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:22:56.880352 9815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:22:56.887426 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 895c8cedf6474ddbaaeb353eea68cbbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=152ca09787ea4fab8f482f6c5ba64f25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:22:56.887941 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c8cedf6474ddbaaeb353eea68cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.887923 10230 tablet_service.cc:1467] Processing CreateTablet for tablet 895c8cedf6474ddbaaeb353eea68cbbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=152ca09787ea4fab8f482f6c5ba64f25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:22:56.888411 10230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c8cedf6474ddbaaeb353eea68cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.894116 10091 tablet_service.cc:1467] Processing CreateTablet for tablet 895c8cedf6474ddbaaeb353eea68cbbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=152ca09787ea4fab8f482f6c5ba64f25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:22:56.894649 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c8cedf6474ddbaaeb353eea68cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:56.895162 10319 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: Bootstrap starting.
I20260928 05:22:56.896690 10319 tablet_bootstrap.cc:654] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.903465 10321 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: Bootstrap starting.
I20260928 05:22:56.905040 10321 tablet_bootstrap.cc:654] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.908820 10319 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: No bootstrap required, opened a new log
I20260928 05:22:56.908947 10320 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: Bootstrap starting.
I20260928 05:22:56.908984 10319 ts_tablet_manager.cc:1397] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260928 05:22:56.909787 10319 raft_consensus.cc:348] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.910025 10319 raft_consensus.cc:374] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.910104 10319 raft_consensus.cc:729] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f421b0b3c5d474597a7fe044e4181b8, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.910408 10319 consensus_queue.cc:260] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.910719 10320 tablet_bootstrap.cc:654] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:56.910844 10325 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:56.911098 10325 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:56.912235 10319 ts_tablet_manager.cc:1428] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:56.916795 10321 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: No bootstrap required, opened a new log
I20260928 05:22:56.916841 10320 tablet_bootstrap.cc:492] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: No bootstrap required, opened a new log
I20260928 05:22:56.916962 10321 ts_tablet_manager.cc:1397] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 05:22:56.916991 10320 ts_tablet_manager.cc:1397] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:22:56.917553 10320 raft_consensus.cc:348] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.917789 10320 raft_consensus.cc:374] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.917675 10321 raft_consensus.cc:348] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.917871 10320 raft_consensus.cc:729] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebca9bd56804e06b0ff101a839f25fa, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.917898 10321 raft_consensus.cc:374] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:56.917970 10321 raft_consensus.cc:729] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285cc9bcb7f948479e7d63b976900d21, State: Initialized, Role: FOLLOWER
I20260928 05:22:56.918104 10320 consensus_queue.cc:260] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.918211 10321 consensus_queue.cc:260] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:56.918591 10326 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:56.918481 10339 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:56.918841 10339 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:56.918849 10326 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:56.919349 10320 ts_tablet_manager.cc:1428] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:22:56.919870 10321 ts_tablet_manager.cc:1428] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260928 05:22:56.968283 10162 tablet.cc:2369] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:22:56.976665 10023 tablet.cc:2369] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:22:57.087877 10326 raft_consensus.cc:482] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:22:57.088120 10326 raft_consensus.cc:504] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:57.088933 10326 leader_election.cc:288] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629), 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165)
I20260928 05:22:57.089440 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c8cedf6474ddbaaeb353eea68cbbf" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8" is_pre_election: true
I20260928 05:22:57.089494 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c8cedf6474ddbaaeb353eea68cbbf" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebca9bd56804e06b0ff101a839f25fa" is_pre_election: true
I20260928 05:22:57.089726 10250 raft_consensus.cc:2454] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 0.
I20260928 05:22:57.089768 9977 raft_consensus.cc:2454] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 0.
I20260928 05:22:57.090117 10044 leader_election.cc:302] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285cc9bcb7f948479e7d63b976900d21, 7f421b0b3c5d474597a7fe044e4181b8; no voters:
I20260928 05:22:57.090425 10326 raft_consensus.cc:2781] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:22:57.090539 10326 raft_consensus.cc:482] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:22:57.090644 10326 raft_consensus.cc:3037] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:57.092551 10326 raft_consensus.cc:504] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:57.093077 10326 leader_election.cc:288] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 election: Requested vote from peers bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629), 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165)
I20260928 05:22:57.093510 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c8cedf6474ddbaaeb353eea68cbbf" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebca9bd56804e06b0ff101a839f25fa"
I20260928 05:22:57.093571 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c8cedf6474ddbaaeb353eea68cbbf" candidate_uuid: "285cc9bcb7f948479e7d63b976900d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f421b0b3c5d474597a7fe044e4181b8"
I20260928 05:22:57.093737 9977 raft_consensus.cc:3037] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:57.093789 10250 raft_consensus.cc:3037] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:57.096493 10250 raft_consensus.cc:2454] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 1.
I20260928 05:22:57.096699 9977 raft_consensus.cc:2454] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285cc9bcb7f948479e7d63b976900d21 in term 1.
I20260928 05:22:57.096920 10044 leader_election.cc:302] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285cc9bcb7f948479e7d63b976900d21, 7f421b0b3c5d474597a7fe044e4181b8; no voters:
I20260928 05:22:57.097167 10326 raft_consensus.cc:2781] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:57.097319 10326 raft_consensus.cc:686] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [term 1 LEADER]: Becoming Leader. State: Replica: 285cc9bcb7f948479e7d63b976900d21, State: Running, Role: LEADER
I20260928 05:22:57.097563 10326 consensus_queue.cc:237] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } }
I20260928 05:22:57.099234 10352 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: TxnStatusTablet state changed. Reason: New leader 285cc9bcb7f948479e7d63b976900d21. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.099611 10352 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:22:57.100293 10364 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:22:57.100521 10364 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:22:57.100745 9814 catalog_manager.cc:5495] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } health_report { overall_health: UNKNOWN } } }
I20260928 05:22:57.142133 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.134598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.8.124.194:43717)
0928 05:22:57.135312 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.135318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.135335 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:57.135460 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.135465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.135571 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.135709 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:57.136105 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.137236 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.138998 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.139730 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.139858 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.140493 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:57.141425 (+ 932us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:57.141574 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.141728 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.141821 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260928 05:22:57.142143 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.134488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43717 (local address 127.0.0.1:58312)
0928 05:22:57.134870 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:57.134892 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:57.135030 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:57.135749 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:57.135755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:57.135769 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:22:57.135964 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:57.135975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.137382 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.137387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:57.138854 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:57.138869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.139974 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.139980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:57.140197 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.140293 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:22:57.141532 (+ 1239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:22:57.141553 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:22:57.141686 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 05:22:57.141839 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260928 05:22:57.147056 10250 raft_consensus.cc:1261] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Refusing update from remote peer 285cc9bcb7f948479e7d63b976900d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:57.147403 9977 raft_consensus.cc:1261] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Refusing update from remote peer 285cc9bcb7f948479e7d63b976900d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:57.147833 10326 consensus_queue.cc:1035] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:57.148458 10326 consensus_queue.cc:1035] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:57.160715 10339 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: TxnStatusTablet state changed. Reason: New leader 285cc9bcb7f948479e7d63b976900d21. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.160979 10339 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:57.160923 10325 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: TxnStatusTablet state changed. Reason: New leader 285cc9bcb7f948479e7d63b976900d21. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.161170 10325 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:57.165139 10352 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.165401 10352 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:22:57.168185 10339 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.168421 10339 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P bebca9bd56804e06b0ff101a839f25fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:57.170435 10326 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.170753 10326 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 285cc9bcb7f948479e7d63b976900d21: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:22:57.172376 10335 mvcc.cc:205] Tried to move back new op lower bound from 7334186914385276928 to 7334186914193903616. Current Snapshot: MvccSnapshot[applied={T|T < 7334186914385276928}]
I20260928 05:22:57.172657 10325 tablet_replica.cc:439] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "285cc9bcb7f948479e7d63b976900d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 } } peers { permanent_uuid: "285cc9bcb7f948479e7d63b976900d21" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43717 } } peers { permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 } } }
I20260928 05:22:57.172873 10325 tablet_replica.cc:442] T 895c8cedf6474ddbaaeb353eea68cbbf P 7f421b0b3c5d474597a7fe044e4181b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:22:57.186062 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.177591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.8.124.194:43717)
0928 05:22:57.178444 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.178453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.178470 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:22:57.178527 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.178532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.178773 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.179269 (+ 496us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:57.180314 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.181456 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.183959 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.184650 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.184788 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.185309 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:57.185445 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:57.185527 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.185693 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.185771 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260928 05:22:57.201979 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.196718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:53067 (local address 127.8.124.194:43717)
0928 05:22:57.196858 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.196866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.197467 (+ 601us) server_negotiation.cc:407] Connection header received
0928 05:22:57.197653 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.197658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.197712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.197785 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:22:57.198253 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.198970 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.200362 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.200782 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.200881 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.201285 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:22:57.201384 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:22:57.201458 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.201587 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.201676 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 05:22:57.202239 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.196802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43717 (local address 127.8.124.194:53067)
0928 05:22:57.197361 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:57.197382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:57.197517 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:57.197806 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:57.197810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:57.197820 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:22:57.198099 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:57.198108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.199080 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.199084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:57.200239 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:57.200251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.200947 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.200952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:57.201063 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.201111 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:22:57.201471 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:22:57.201481 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:22:57.201551 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:22:57.201699 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":282,"threads_started":1}
I20260928 05:22:57.211930 10250 raft_consensus.cc:1261] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Refusing update from remote peer 285cc9bcb7f948479e7d63b976900d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:57.212024 9977 raft_consensus.cc:1261] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Refusing update from remote peer 285cc9bcb7f948479e7d63b976900d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:22:57.212481 10352 consensus_queue.cc:1035] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebca9bd56804e06b0ff101a839f25fa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:57.212757 10326 consensus_queue.cc:1035] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f421b0b3c5d474597a7fe044e4181b8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:22:57.274993 9815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:57.275255 9815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:22:57.278854 9815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98df7dc3fd3d448ca8722cd517d26fab: Sending DeleteTablet for 3 replicas of tablet 57407bd9fc344bc1b96a71452dbf65c9
I20260928 05:22:57.279608 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:57 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:51876
I20260928 05:22:57.279726 10230 tablet_service.cc:1514] Processing DeleteTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:57 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39348
I20260928 05:22:57.279870 10344 tablet_replica.cc:330] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: stopping tablet replica
I20260928 05:22:57.280002 10345 tablet_replica.cc:330] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: stopping tablet replica
I20260928 05:22:57.280028 10344 raft_consensus.cc:2229] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:57.280176 10344 raft_consensus.cc:2258] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:57.280184 10345 raft_consensus.cc:2229] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:22:57.280354 10345 raft_consensus.cc:2258] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:57.281277 10344 ts_tablet_manager.cc:1905] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:57.281410 10345 ts_tablet_manager.cc:1905] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:57.281443 10091 tablet_service.cc:1514] Processing DeleteTablet for tablet 57407bd9fc344bc1b96a71452dbf65c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:22:57 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:58296
I20260928 05:22:57.287987 10344 ts_tablet_manager.cc:1918] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:22:57.288136 10344 log.cc:1196] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/57407bd9fc344bc1b96a71452dbf65c9
I20260928 05:22:57.288295 10345 ts_tablet_manager.cc:1918] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:22:57.288435 10345 log.cc:1196] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/57407bd9fc344bc1b96a71452dbf65c9
I20260928 05:22:57.288522 10344 ts_tablet_manager.cc:1939] T 57407bd9fc344bc1b96a71452dbf65c9 P bebca9bd56804e06b0ff101a839f25fa: Deleting consensus metadata
I20260928 05:22:57.288866 10345 ts_tablet_manager.cc:1939] T 57407bd9fc344bc1b96a71452dbf65c9 P 7f421b0b3c5d474597a7fe044e4181b8: Deleting consensus metadata
I20260928 05:22:57.289556 9799 catalog_manager.cc:4841] TS bebca9bd56804e06b0ff101a839f25fa (127.8.124.193:43629): tablet 57407bd9fc344bc1b96a71452dbf65c9 (table test-table [id=355bd4225a6445219744da2bd4c9670d]) successfully deleted
I20260928 05:22:57.289768 10346 tablet_replica.cc:330] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: stopping tablet replica
I20260928 05:22:57.289995 10346 raft_consensus.cc:2229] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:22:57.290589 10346 raft_consensus.cc:2258] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:22:57.291203 9801 catalog_manager.cc:4841] TS 7f421b0b3c5d474597a7fe044e4181b8 (127.8.124.195:42165): tablet 57407bd9fc344bc1b96a71452dbf65c9 (table test-table [id=355bd4225a6445219744da2bd4c9670d]) successfully deleted
I20260928 05:22:57.291741 10346 ts_tablet_manager.cc:1905] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:22:57.298820 10346 ts_tablet_manager.cc:1918] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:22:57.298976 10346 log.cc:1196] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/57407bd9fc344bc1b96a71452dbf65c9
I20260928 05:22:57.299410 10346 ts_tablet_manager.cc:1939] T 57407bd9fc344bc1b96a71452dbf65c9 P 285cc9bcb7f948479e7d63b976900d21: Deleting consensus metadata
I20260928 05:22:57.300442 9801 catalog_manager.cc:4841] TS 285cc9bcb7f948479e7d63b976900d21 (127.8.124.194:43717): tablet 57407bd9fc344bc1b96a71452dbf65c9 (table test-table [id=355bd4225a6445219744da2bd4c9670d]) successfully deleted
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:22:57.309131 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.294395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.8.124.193:43629)
0928 05:22:57.294558 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.294565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.295180 (+ 615us) server_negotiation.cc:407] Connection header received
0928 05:22:57.295359 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.295365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.295434 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.295522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:57.296889 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.297555 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.298831 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.299418 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.299552 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.303689 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:57.303704 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:57.303721 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:57.303753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:57.306563 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.307243 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.307251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.307254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.307391 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.307731 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.307736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.307738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.308094 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:57.308380 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.308621 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.308822 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 05:22:57.309830 10017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51906
I20260928 05:22:57.325830 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.312584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.8.124.254:36529)
0928 05:22:57.312789 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.312795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.312999 (+ 204us) server_negotiation.cc:407] Connection header received
0928 05:22:57.313158 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.313161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.313221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.313334 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:57.314911 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.315634 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.317399 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.317855 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.317948 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.320775 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:57.320782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:57.320786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:57.320820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:57.323310 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.324058 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.324061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.324064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.324184 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.324528 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.324532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.324551 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.324834 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:57.325060 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.325353 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.325511 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 05:22:57.326344 9815 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60168
I20260928 05:22:57.328330 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9886
I20260928 05:22:57.346432 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.329155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.8.124.193:43629)
0928 05:22:57.329868 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.329875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.334173 (+ 4298us) server_negotiation.cc:407] Connection header received
0928 05:22:57.334357 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.334361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.334460 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.334544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:57.336075 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.336805 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.338538 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.338921 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.339012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.341540 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:57.341546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:57.341549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:57.341581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:57.343956 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.344570 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.344576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.344578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.344700 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.345030 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.345035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.345038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.345291 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:57.345554 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.345927 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.346143 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
W20260928 05:22:57.347023 10017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51912
W20260928 05:22:57.347510 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:57.347687 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9886
I20260928 05:22:57.373549 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10026
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:22:57.388418 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.374322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.8.124.194:43717)
0928 05:22:57.374504 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.374510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.374528 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:22:57.374776 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.374784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.374850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.374938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:57.376031 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.376827 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.378073 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.378528 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.378630 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.382912 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:57.382923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:57.382936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:57.382971 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:57.385901 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.386638 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.386646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.386648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.386761 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.387108 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.387111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.387113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.387442 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:57.387681 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.387963 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.388112 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 05:22:57.389149 10156 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58328
W20260928 05:22:57.389816 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:57.389973 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10026
I20260928 05:22:57.404910 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10165
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[9770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:22:57.418490 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:57.405551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.8.124.195:42165)
0928 05:22:57.406007 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:57.406011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:57.406027 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:22:57.406077 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:57.406081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:57.406151 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:57.406245 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:57.407337 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.408060 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.409185 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:57.409575 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:57.409726 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:57.413511 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:57.413527 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:57.413544 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:57.413580 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:57.416072 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.416633 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.416640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.416642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.416769 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:57.417051 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:57.417055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:57.417057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:57.417424 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:57.417699 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:57.418001 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 05:22:57.418161 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
W20260928 05:22:57.419108 10295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39362
W20260928 05:22:57.419675 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10165: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:22:57.419818 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10165
I20260928 05:22:57.432857 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 9783
I20260928 05:22:57.433830 9855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:22:57.543593 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 9783
2026-09-28T05:22:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4202 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[10386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[10386](info): set up 2 sockets
Sep 28 05:22:57 dist-test-slave-515x krb5kdc[10386](info): commencing operation
krb5kdc: starting...
W20260928 05:22:58.549638 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:22:58 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:22:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:22:58.715022 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36481
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:59.018270 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:59.018548 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:59.018673 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:59.026057 10399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:22:59.026163 10399 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:22:59.026232 10399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:59.026307 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:59.026371 10399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:22:59.026444 10399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:22:59.035988 10399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36481
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:59.037186 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:59.038558 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:59.049937 10404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:59.050024 10407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:59.050079 10405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:59.050792 10399 server_base.cc:1034] running on GCE node
I20260928 05:22:59.051528 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:59.052901 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:59.054096 10399 hybrid_clock.cc:648] HybridClock initialized: now 1790572979054037 us; error 42 us; skew 500 ppm
I20260928 05:22:59.054404 10399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:59 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572979, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:59.057837 10399 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:22:59.059218 10399 webserver.cc:460] Webserver started at http://127.8.124.254:45589/ using document root <none> and password file <none>
I20260928 05:22:59.059880 10399 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:59.059962 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:59.060204 10399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:59.063256 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "8760fea48a7223fc79f02d26eff01790"
server_key_iv: "549ea3cf50d8d9112cf4cdf2cb3080bf"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.063985 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "8760fea48a7223fc79f02d26eff01790"
server_key_iv: "549ea3cf50d8d9112cf4cdf2cb3080bf"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.069001 10399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:22:59.072324 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:59.073557 10399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:22:59.073719 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "8760fea48a7223fc79f02d26eff01790"
server_key_iv: "549ea3cf50d8d9112cf4cdf2cb3080bf"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.073889 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:59.095788 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:59.099897 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:59.100108 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:59.119097 10399 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:36481
I20260928 05:22:59.119112 10475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:36481 every 8 connection(s)
I20260928 05:22:59.120543 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:22:59.124327 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:22:59.130016 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10399
I20260928 05:22:59.130256 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:22:59.130891 8691 external_mini_cluster.cc:1428] Setting key ad4ad48ea05809d653da070cc5da3dba
I20260928 05:22:59.134836 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Bootstrap starting.
I20260928 05:22:59.138463 10476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Neither blocks nor log segments found. Creating new log.
I20260928 05:22:59.139448 10476 log.cc:824] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Log is configured to *not* fsync() on all Append() calls
I20260928 05:22:59.142385 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: No bootstrap required, opened a new log
Sep 28 05:22:59 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:59.150574 10476 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } }
I20260928 05:22:59.150945 10476 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:22:59.151098 10476 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5648cd2cb2f4ee4b4b335e8477de7ce, State: Initialized, Role: FOLLOWER
I20260928 05:22:59.151804 10476 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } }
I20260928 05:22:59.152046 10476 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:22:59.152174 10476 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:22:59.152336 10476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:22:59.153131 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:59.132866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.8.124.254:36481)
0928 05:22:59.133651 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:59.133666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:59.133708 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:22:59.135352 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:59.135378 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:59.135787 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:59.136115 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:59.138005 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.139384 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.140854 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.141507 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.141683 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:59.146147 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:59.146154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:59.146169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:59.146203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:59.149149 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.149832 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.149838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.149845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.149997 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.150373 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.150377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.150379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.150887 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:59.151157 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:59.151607 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 05:22:59.152012 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":341,"threads_started":1}
I20260928 05:22:59.154663 10476 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } }
I20260928 05:22:59.155133 10476 leader_election.cc:302] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5648cd2cb2f4ee4b4b335e8477de7ce; no voters:
I20260928 05:22:59.155685 10476 leader_election.cc:288] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:22:59.155791 10481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:22:59.156425 10481 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [term 1 LEADER]: Becoming Leader. State: Replica: a5648cd2cb2f4ee4b4b335e8477de7ce, State: Running, Role: LEADER
I20260928 05:22:59.156960 10481 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } }
I20260928 05:22:59.157847 10476 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:22:59.159650 10482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } } }
I20260928 05:22:59.159968 10482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:59.163370 10483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5648cd2cb2f4ee4b4b335e8477de7ce. Latest consensus state: current_term: 1 leader_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5648cd2cb2f4ee4b4b335e8477de7ce" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36481 } } }
I20260928 05:22:59.163683 10483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce [sys.catalog]: This master's current role is: LEADER
I20260928 05:22:59.164294 10487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:22:59.168936 10487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:22:59.183207 10487 catalog_manager.cc:1347] Generated new cluster ID: a00fc239eedd42cd8c8d3dfc0d3ae7e7
I20260928 05:22:59.183331 10487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:22:59.205327 10487 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:22:59.207221 10487 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:22:59.224474 10487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Generated new TSK 0
I20260928 05:22:59.225286 10487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:22:59.281013 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36481
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:22:59.620419 10502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:22:59.620699 10502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:22:59.620800 10502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:22:59.627835 10502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:22:59.627934 10502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:22:59.628028 10502 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:22:59.628079 10502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:22:59.628146 10502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:22:59.637491 10502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:22:59.638661 10502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:22:59.639878 10502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:22:59.651808 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:59.651860 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:22:59.651883 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:22:59.653128 10502 server_base.cc:1034] running on GCE node
I20260928 05:22:59.653812 10502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:22:59.654709 10502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:22:59.655907 10502 hybrid_clock.cc:648] HybridClock initialized: now 1790572979655859 us; error 43 us; skew 500 ppm
I20260928 05:22:59.656201 10502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:22:59 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572979, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:22:59.659242 10502 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:22:59.660629 10502 webserver.cc:460] Webserver started at http://127.8.124.193:34383/ using document root <none> and password file <none>
I20260928 05:22:59.661252 10502 fs_manager.cc:359] Metadata directory not provided
I20260928 05:22:59.661342 10502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:22:59.661579 10502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:22:59.664688 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "c626403d460c4018b8913140cfe0b351"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "14f3a68d4b9c1905f713b839e5445073"
server_key_iv: "185a1f0a970baa72ff3a71c349cc952a"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.665370 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "c626403d460c4018b8913140cfe0b351"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "14f3a68d4b9c1905f713b839e5445073"
server_key_iv: "185a1f0a970baa72ff3a71c349cc952a"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.670300 10502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:22:59.673535 10517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:59.674816 10502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:22:59.674943 10502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "c626403d460c4018b8913140cfe0b351"
format_stamp: "Formatted at 2026-09-28 05:22:59 on dist-test-slave-515x"
server_key: "14f3a68d4b9c1905f713b839e5445073"
server_key_iv: "185a1f0a970baa72ff3a71c349cc952a"
server_key_version: "encryptionkey@0"
I20260928 05:22:59.675092 10502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:22:59.697006 10502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:22:59.700711 10502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:22:59.700939 10502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:22:59.703477 10502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:22:59.703569 10502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:22:59.703671 10502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:22:59.703752 10502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:22:59 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572979, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:22:59.726023 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:59.703929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:35043 (local address 127.8.124.254:36481)
0928 05:22:59.704581 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:59.704587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:59.705638 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 05:22:59.707645 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:59.707651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:59.707753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:59.707856 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:59.710349 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.711289 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.714129 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.714604 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.714734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:59.719781 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:59.719789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:59.719794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:59.719836 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:59.722707 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.723390 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.723395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.723400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.723541 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.723948 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.723953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.723956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.724350 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:59.724481 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:59.725433 (+ 952us) server_negotiation.cc:299] Negotiation successful
0928 05:22:59.725680 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":422,"threads_started":1}
I20260928 05:22:59.727351 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:59.704601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.193:35043)
0928 05:22:59.705449 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:59.705493 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:59.707357 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:59.708225 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:59.708237 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:59.709106 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:59.710113 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:59.710145 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.711457 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.711464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:59.713961 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:59.713981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.714871 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.714879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:59.715281 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:59.716133 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:59.716171 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:59.719507 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:59.722841 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:59.722850 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:59.722867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:59.723265 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:59.723639 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:59.723644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:59.723649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:59.723840 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:59.724547 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:59.724555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:59.725008 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 05:22:59.725297 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 05:22:59.725848 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":333,"threads_started":1}
I20260928 05:22:59.746593 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:59.733338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:45561 (local address 127.8.124.254:36481)
0928 05:22:59.733526 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:22:59.733533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:22:59.734111 (+ 578us) server_negotiation.cc:407] Connection header received
0928 05:22:59.734292 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:22:59.734298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:22:59.734385 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:22:59.734495 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:22:59.735630 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.736668 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.738421 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.738944 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.739069 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:59.741190 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:22:59.741194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:22:59.741198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:22:59.741227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:22:59.744012 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.744643 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.744649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.744653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.744771 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:22:59.745078 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:22:59.745083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:22:59.745087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:22:59.745349 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:22:59.745475 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:22:59.745901 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 05:22:59.746249 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:22:59.746690 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:22:59.733355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.193:45561)
0928 05:22:59.734003 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 05:22:59.734023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:22:59.734172 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:22:59.734513 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:22:59.734516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:22:59.735011 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:22:59.735437 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:59.735448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.736827 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.736831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:59.738221 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:22:59.738247 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:22:59.739144 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:22:59.739151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:22:59.739340 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:22:59.739938 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:22:59.739960 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:22:59.741012 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:22:59.744126 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:59.744132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:59.744136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:59.744544 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:59.744866 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:22:59.744871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:22:59.744875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:22:59.744988 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:22:59.745453 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:22:59.745458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:22:59.745672 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 05:22:59.745807 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 05:22:59.746264 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":343,"threads_started":1}
I20260928 05:22:59.749437 10502 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33913
I20260928 05:22:59.749482 10637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33913 every 8 connection(s)
I20260928 05:22:59.750887 10502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:22:59.752640 10638 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36481
I20260928 05:22:59.752883 10638 heartbeater.cc:464] Registering TS with master...
I20260928 05:22:59.753448 10638 heartbeater.cc:511] Master 127.8.124.254:36481 requested a full tablet report, sending...
I20260928 05:22:59.755194 10431 ts_manager.cc:194] Registered new tserver with Master: c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913)
I20260928 05:22:59.756536 10431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:35043
I20260928 05:22:59.758949 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10502
I20260928 05:22:59.759116 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:22:59.759694 8691 external_mini_cluster.cc:1428] Setting key 3ed98ca761b6332fdd399213cf6e7a59
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:22:59.794608 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36481
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:00.098273 10642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:00.098552 10642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:00.098672 10642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:00.105818 10642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:00.105921 10642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:00.106030 10642 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:23:00.106091 10642 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:23:00.106196 10642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:00.115717 10642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:00.116964 10642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:00.118326 10642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:00.130743 10648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:00.131302 10647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:00.131362 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:00.131592 10642 server_base.cc:1034] running on GCE node
I20260928 05:23:00.132277 10642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:00.133177 10642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:00.134368 10642 hybrid_clock.cc:648] HybridClock initialized: now 1790572980134311 us; error 52 us; skew 500 ppm
I20260928 05:23:00.134663 10642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:00.137631 10642 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:00.138989 10642 webserver.cc:460] Webserver started at http://127.8.124.194:41461/ using document root <none> and password file <none>
I20260928 05:23:00.139635 10642 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:00.139750 10642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:00.140018 10642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:00.142825 10642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "09049f079f2e4189918a77b3ddf2bc0e"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "4f0fd185b1bde754b563bb46a9c28f4d"
server_key_iv: "d9e216a20d2b3b7c10bd93dd57bebc7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.143505 10642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "09049f079f2e4189918a77b3ddf2bc0e"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "4f0fd185b1bde754b563bb46a9c28f4d"
server_key_iv: "d9e216a20d2b3b7c10bd93dd57bebc7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.148335 10642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:00.151548 10657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:00.152808 10642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:00.152944 10642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "09049f079f2e4189918a77b3ddf2bc0e"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "4f0fd185b1bde754b563bb46a9c28f4d"
server_key_iv: "d9e216a20d2b3b7c10bd93dd57bebc7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.153105 10642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:00.164662 10642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:00.168924 10642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:00.169139 10642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:00.171559 10642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:00.171659 10642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:00.171774 10642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:00.171845 10642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:00.193620 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.172870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.194:55643)
0928 05:23:00.173738 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.173791 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.175674 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.176985 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.176996 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.177942 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:00.178920 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.178955 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.180132 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.180136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.181431 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.181443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.182491 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.182499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.182796 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.183597 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:00.183618 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:00.186204 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:00.189144 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.189153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.189170 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.189582 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.189952 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.189957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.189961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.190138 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.190737 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:00.190748 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:00.191189 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.191520 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.192062 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":338,"threads_started":1}
I20260928 05:23:00.194360 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.172329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:55643 (local address 127.8.124.254:36481)
0928 05:23:00.172507 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.172513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.173901 (+ 1388us) server_negotiation.cc:407] Connection header received
0928 05:23:00.176047 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.176053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.176150 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.176447 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.179139 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.179978 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.181563 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.182079 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.182207 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.186415 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.186420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.186423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.186453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.189026 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.189731 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.189738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.189741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.189851 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.190262 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.190267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.190270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.190610 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.190746 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.193839 (+ 3093us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.194051 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:00.214983 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.199599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.194:52079)
0928 05:23:00.200981 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.200999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.201143 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.201440 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.201444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.201941 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:00.202377 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.202391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.203584 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.203590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.205528 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.205543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.206391 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.206399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.206582 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.207201 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:00.207224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:00.208264 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:00.211782 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.211788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.211791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.212351 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.212745 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.212750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.212753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.212906 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.213546 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:00.213551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:00.213810 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.213990 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.214418 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":534,"threads_started":1}
I20260928 05:23:00.215935 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.199945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:52079 (local address 127.8.124.254:36481)
0928 05:23:00.200107 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.200112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.201080 (+ 968us) server_negotiation.cc:407] Connection header received
0928 05:23:00.201251 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.201256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.201331 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.201405 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.202513 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.203434 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.205702 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.206197 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.206321 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.208464 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.208469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.208472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.208502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.211603 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.212479 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.212484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.212486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.212630 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.213052 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.213058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.213061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.213430 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.213579 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.215337 (+ 1758us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.215593 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:23:00.222268 10642 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:43967
I20260928 05:23:00.222318 10776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:43967 every 8 connection(s)
I20260928 05:23:00.223429 10642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:00.225488 10777 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36481
I20260928 05:23:00.225735 10777 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:00.226418 10777 heartbeater.cc:511] Master 127.8.124.254:36481 requested a full tablet report, sending...
I20260928 05:23:00.227723 10431 ts_manager.cc:194] Registered new tserver with Master: 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967)
I20260928 05:23:00.228466 10431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:55643
I20260928 05:23:00.230201 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10642
I20260928 05:23:00.230343 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:00.230765 8691 external_mini_cluster.cc:1428] Setting key 6525fbaf9b97cd7e9f49916c83e8a567
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:00.266665 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36481
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:00.579417 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:00.579694 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:00.579851 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:00.587061 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:00.587162 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:00.587273 10781 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:23:00.587334 10781 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:23:00.587450 10781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:00.597416 10781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:36481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:00.598677 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:00.599963 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:00.612703 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:00.612670 10789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:00.612723 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:00.613375 10781 server_base.cc:1034] running on GCE node
I20260928 05:23:00.614120 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:00.615039 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:00.616238 10781 hybrid_clock.cc:648] HybridClock initialized: now 1790572980616183 us; error 40 us; skew 500 ppm
I20260928 05:23:00.616537 10781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:00.619561 10781 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:00.620993 10781 webserver.cc:460] Webserver started at http://127.8.124.195:34655/ using document root <none> and password file <none>
I20260928 05:23:00.621742 10781 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:00.621846 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:00.622094 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:00.624825 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "ac1a597683b745ecab8c4d13bbe88f09"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "83aa425d2029345c371f3c66369f23cc"
server_key_iv: "fc97b5a2e5aee68284b49d9cec82195a"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.625486 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "ac1a597683b745ecab8c4d13bbe88f09"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "83aa425d2029345c371f3c66369f23cc"
server_key_iv: "fc97b5a2e5aee68284b49d9cec82195a"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.630321 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:23:00.633488 10796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:00.634814 10781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:00.634955 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "ac1a597683b745ecab8c4d13bbe88f09"
format_stamp: "Formatted at 2026-09-28 05:23:00 on dist-test-slave-515x"
server_key: "83aa425d2029345c371f3c66369f23cc"
server_key_iv: "fc97b5a2e5aee68284b49d9cec82195a"
server_key_version: "encryptionkey@0"
I20260928 05:23:00.635121 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:00.651818 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:00.656038 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:00.656306 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:00.659047 10781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:00.659158 10781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:00.659283 10781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:00.659430 10781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:00.680233 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.659463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:57747 (local address 127.8.124.254:36481)
0928 05:23:00.659605 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.659610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.661098 (+ 1488us) server_negotiation.cc:407] Connection header received
0928 05:23:00.663228 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.663235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.663313 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.663397 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.665434 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.666617 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.668811 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.669379 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.669506 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.674185 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.674190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.674193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.674248 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.677112 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.677971 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.677977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.677981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.678106 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.678528 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.678533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.678537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.678810 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.678952 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.679682 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.679892 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:23:00.681188 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.660192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.195:57747)
0928 05:23:00.660955 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.660992 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.662925 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.663651 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.663664 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.664401 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:00.665210 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.665240 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.666796 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.666805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.668678 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.668690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.669805 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.669811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.670059 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.671025 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:00.671055 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:00.673970 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:00.677233 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.677241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.677258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.677872 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.678204 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.678209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.678213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.678420 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.678943 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:00.678950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:00.679307 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.679547 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.680037 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":320,"threads_started":1}
I20260928 05:23:00.700392 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.686888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:41803 (local address 127.8.124.254:36481)
0928 05:23:00.687105 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.687109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.687194 (+ 85us) server_negotiation.cc:407] Connection header received
0928 05:23:00.688220 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.688225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.688278 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.688354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.689451 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.690186 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.692050 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.692629 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.692740 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.694888 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.694896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.694902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.694944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.698021 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.698649 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.698653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.698655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.698772 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.699086 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.699091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.699095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.699344 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.699475 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.699834 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.700034 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 05:23:00.700469 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.686494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.8.124.195:41803)
0928 05:23:00.687087 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.687107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.688093 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.688401 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.688406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.688967 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:00.689325 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.689337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.690324 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.690328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.691907 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.691927 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.692813 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.692821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.692973 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.693645 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:00.693672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:00.694696 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:00.698143 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.698148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.698155 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.698544 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.698878 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.698881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.698884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.698993 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.699450 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:00.699455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:00.699614 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.699724 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.700033 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":297,"threads_started":1}
I20260928 05:23:00.710728 10781 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:38463
I20260928 05:23:00.710757 10915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:38463 every 8 connection(s)
I20260928 05:23:00.711838 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:00.712904 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 10781
I20260928 05:23:00.713068 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:00.713562 8691 external_mini_cluster.cc:1428] Setting key a98068770a031e761d35164c1cb509e6
I20260928 05:23:00.713758 10916 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36481
I20260928 05:23:00.713958 10916 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:00.714531 10916 heartbeater.cc:511] Master 127.8.124.254:36481 requested a full tablet report, sending...
I20260928 05:23:00.715883 10430 ts_manager.cc:194] Registered new tserver with Master: ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:00.716761 10430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:57747
I20260928 05:23:00.728260 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 05:23:00.759043 10638 heartbeater.cc:503] Master 127.8.124.254:36481 was elected leader, sending a full tablet report...
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:00.772176 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.760023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.8.124.254:36481)
0928 05:23:00.760197 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.760203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.760484 (+ 281us) server_negotiation.cc:407] Connection header received
0928 05:23:00.760658 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.760663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.760736 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.760853 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.762099 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.762806 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.763928 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.764318 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.764414 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.767737 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.767742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.767745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.767775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.770216 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.770722 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.770726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.770728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.770850 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.771121 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.771124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.771126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.771377 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.771539 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.771747 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.771923 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:23:00.776319 10430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:00.783125 10430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:00.803828 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.795208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.8.124.194:43967)
0928 05:23:00.795775 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.795784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.795814 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:23:00.795911 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.795920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.796214 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.797150 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:00.797342 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.798950 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.801425 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.802737 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.802875 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.803082 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.803399 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.803551 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":245,"threads_started":1}
I20260928 05:23:00.804003 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.794494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.0.0.1:44834)
0928 05:23:00.795294 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.795345 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.795502 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.796462 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.796470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.796503 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:00.796951 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.796963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.799081 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.799088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.801280 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.801300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.802960 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.802967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.803187 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.803232 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.803342 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.803531 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":30,"thread_start_us":352,"threads_started":1}
I20260928 05:23:00.804705 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.795674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.8.124.195:38463)
0928 05:23:00.796415 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.796423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.796885 (+ 462us) server_negotiation.cc:407] Connection header received
0928 05:23:00.796980 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.796988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.797233 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.797404 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:00.798304 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.799557 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.801986 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.803401 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.803518 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.803721 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.804247 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.804413 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260928 05:23:00.805682 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.794621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38463 (local address 127.0.0.1:34246)
0928 05:23:00.796613 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.796634 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.796759 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.797680 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.797685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.797707 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:00.798091 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.798103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.799681 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.799686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.801804 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.801818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.803889 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.803895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.804062 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.804094 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.804873 (+ 779us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.805296 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1873,"mutex_wait_us":4,"thread_start_us":1960,"threads_started":1}
I20260928 05:23:00.807206 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.796793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33913 (local address 127.0.0.1:43166)
0928 05:23:00.797294 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.797315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.797411 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.800614 (+ 3203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.800619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.800644 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:00.801103 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.801114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.802928 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.802937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.804939 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.804951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.806530 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.806536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.806697 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.806730 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.806802 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.806918 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":660,"threads_started":1}
I20260928 05:23:00.807857 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.797304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.8.124.193:33913)
0928 05:23:00.800038 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.800047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.800072 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:23:00.800162 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.800170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.800394 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.800582 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:00.801305 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.802745 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.805109 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.806356 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.806473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.807489 (+ 1016us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.807550 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.807657 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2580,"thread_start_us":2463,"threads_started":1}
I20260928 05:23:00.807618 10707 tablet_service.cc:1467] Processing CreateTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d (DEFAULT_TABLE table=test-table [id=addee9da48a4447b801da9769c9ae644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:00.809031 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48b568b954840ab8a3ff4a5d6dbb80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.810432 10846 tablet_service.cc:1467] Processing CreateTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d (DEFAULT_TABLE table=test-table [id=addee9da48a4447b801da9769c9ae644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:00.811113 10568 tablet_service.cc:1467] Processing CreateTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d (DEFAULT_TABLE table=test-table [id=addee9da48a4447b801da9769c9ae644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:00.811668 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48b568b954840ab8a3ff4a5d6dbb80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.812373 10568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48b568b954840ab8a3ff4a5d6dbb80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.821864 10935 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Bootstrap starting.
I20260928 05:23:00.825544 10935 tablet_bootstrap.cc:654] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:00.825757 10937 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Bootstrap starting.
I20260928 05:23:00.827221 10935 log.cc:824] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:00.828011 10936 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Bootstrap starting.
I20260928 05:23:00.829017 10937 tablet_bootstrap.cc:654] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:00.830147 10937 log.cc:824] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:00.830963 10936 tablet_bootstrap.cc:654] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:00.831288 10935 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: No bootstrap required, opened a new log
I20260928 05:23:00.831607 10935 ts_tablet_manager.cc:1397] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260928 05:23:00.832046 10936 log.cc:824] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:00.832931 10937 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: No bootstrap required, opened a new log
I20260928 05:23:00.833202 10937 ts_tablet_manager.cc:1397] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:23:00.834239 10936 tablet_bootstrap.cc:492] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: No bootstrap required, opened a new log
I20260928 05:23:00.834442 10936 ts_tablet_manager.cc:1397] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:23:00.841017 10936 raft_consensus.cc:348] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.841344 10936 raft_consensus.cc:374] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.841451 10936 raft_consensus.cc:729] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.842060 10936 consensus_queue.cc:260] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.842517 10935 raft_consensus.cc:348] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.842981 10935 raft_consensus.cc:374] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.843103 10935 raft_consensus.cc:729] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.842957 10937 raft_consensus.cc:348] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.843343 10937 raft_consensus.cc:374] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.843492 10937 raft_consensus.cc:729] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c626403d460c4018b8913140cfe0b351, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.843668 10935 consensus_queue.cc:260] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.844012 10937 consensus_queue.cc:260] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.844734 10916 heartbeater.cc:503] Master 127.8.124.254:36481 was elected leader, sending a full tablet report...
I20260928 05:23:00.845428 10935 ts_tablet_manager.cc:1428] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 05:23:00.846812 10937 ts_tablet_manager.cc:1428] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260928 05:23:00.847849 10777 heartbeater.cc:503] Master 127.8.124.254:36481 was elected leader, sending a full tablet report...
I20260928 05:23:00.848573 10936 ts_tablet_manager.cc:1428] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 05:23:00.887989 10942 raft_consensus.cc:482] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:00.888324 10942 raft_consensus.cc:504] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.889943 10942 leader_election.cc:288] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967), c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913)
I20260928 05:23:00.896484 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.890423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33913 (local address 127.8.124.195:36299)
0928 05:23:00.890699 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.890718 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.890857 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.891220 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.891224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.891254 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:00.891495 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.891509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.892677 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.892682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.894484 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.894499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.895866 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.895870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.896018 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.896043 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.896114 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.896223 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1}
I20260928 05:23:00.896538 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.890571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:36299 (local address 127.8.124.193:33913)
0928 05:23:00.890772 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.890777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.890795 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:00.890994 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.891000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.891076 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.891151 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:00.891650 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.892527 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.894627 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.895733 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.895886 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.895960 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.896155 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.896302 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:00.897256 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.890503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:40983 (local address 127.8.124.194:43967)
0928 05:23:00.890705 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.890712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.891823 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 05:23:00.891943 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.891947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.892020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.892113 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:00.892507 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.893204 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.895216 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.896355 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.896510 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.896573 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.896942 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.897044 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 05:23:00.897325 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.890974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.8.124.195:40983)
0928 05:23:00.891734 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.891747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.891836 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.892121 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.892124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.892140 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:00.892375 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.892382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.893356 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.893362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.895028 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.895038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.896612 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.896618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.896787 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.896820 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.896897 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.897037 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":432,"threads_started":1}
I20260928 05:23:00.897778 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48b568b954840ab8a3ff4a5d6dbb80d" candidate_uuid: "ac1a597683b745ecab8c4d13bbe88f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351" is_pre_election: true
I20260928 05:23:00.898254 10588 raft_consensus.cc:2454] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a597683b745ecab8c4d13bbe88f09 in term 0.
I20260928 05:23:00.898252 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48b568b954840ab8a3ff4a5d6dbb80d" candidate_uuid: "ac1a597683b745ecab8c4d13bbe88f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09049f079f2e4189918a77b3ddf2bc0e" is_pre_election: true
I20260928 05:23:00.898699 10732 raft_consensus.cc:2454] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a597683b745ecab8c4d13bbe88f09 in term 0.
I20260928 05:23:00.898901 10798 leader_election.cc:302] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac1a597683b745ecab8c4d13bbe88f09, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:00.899245 10942 raft_consensus.cc:2781] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:00.899384 10942 raft_consensus.cc:482] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:00.899479 10942 raft_consensus.cc:3037] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:00.901556 10942 raft_consensus.cc:504] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.902176 10942 leader_election.cc:288] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [CANDIDATE]: Term 1 election: Requested vote from peers 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967), c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913)
I20260928 05:23:00.902477 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48b568b954840ab8a3ff4a5d6dbb80d" candidate_uuid: "ac1a597683b745ecab8c4d13bbe88f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09049f079f2e4189918a77b3ddf2bc0e"
I20260928 05:23:00.902534 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48b568b954840ab8a3ff4a5d6dbb80d" candidate_uuid: "ac1a597683b745ecab8c4d13bbe88f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351"
I20260928 05:23:00.902627 10732 raft_consensus.cc:3037] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:00.902709 10588 raft_consensus.cc:3037] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:00.904654 10732 raft_consensus.cc:2454] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a597683b745ecab8c4d13bbe88f09 in term 1.
I20260928 05:23:00.904799 10588 raft_consensus.cc:2454] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a597683b745ecab8c4d13bbe88f09 in term 1.
I20260928 05:23:00.904989 10800 leader_election.cc:302] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, ac1a597683b745ecab8c4d13bbe88f09; no voters:
I20260928 05:23:00.905252 10942 raft_consensus.cc:2781] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:00.905661 10942 raft_consensus.cc:686] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 1 LEADER]: Becoming Leader. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Running, Role: LEADER
I20260928 05:23:00.906087 10942 consensus_queue.cc:237] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } }
I20260928 05:23:00.910001 10431 catalog_manager.cc:5495] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195). New cstate: current_term: 1 leader_uuid: "ac1a597683b745ecab8c4d13bbe88f09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:23:00 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572979, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:00.943756 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.931730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36481 (local address 127.0.0.1:35046)
0928 05:23:00.932242 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:00.932256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:00.932392 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:00.932667 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:00.932672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:00.933308 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:00.933548 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.933557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.934498 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.934503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.935444 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:00.935455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.936105 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.936111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:00.936246 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.936804 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:00.936819 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:00.938872 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:00.941497 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.941506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.941512 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.941944 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.942271 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:00.942275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:00.942278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:00.942423 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:00.942890 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:00.942896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:00.943041 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 05:23:00.943218 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 05:23:00.943439 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":4,"thread_start_us":270,"threads_started":1}
I20260928 05:23:00.943804 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:00.931665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.8.124.254:36481)
0928 05:23:00.931812 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:00.931817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:00.932337 (+ 520us) server_negotiation.cc:407] Connection header received
0928 05:23:00.932493 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:00.932497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:00.932557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:00.932632 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:00.933690 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.934389 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.935561 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:00.935950 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:00.936049 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:00.939047 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:00.939052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:00.939056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:00.939084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:00.941388 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.942031 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.942035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.942037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.942158 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:00.942507 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:00.942510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:00.942512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:00.942760 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:00.942877 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:00.943255 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 05:23:00.943543 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":52}
I20260928 05:23:00.948364 10431 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:35046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:23:00.949275 10431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:00.955333 10846 tablet_service.cc:1467] Processing CreateTablet for tablet 86682656c9514a8da95020107f6e563f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52dede414b564529ad0fec2e9c5065d6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:00.955188 10568 tablet_service.cc:1467] Processing CreateTablet for tablet 86682656c9514a8da95020107f6e563f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52dede414b564529ad0fec2e9c5065d6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:00.955329 10707 tablet_service.cc:1467] Processing CreateTablet for tablet 86682656c9514a8da95020107f6e563f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52dede414b564529ad0fec2e9c5065d6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:00.956094 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86682656c9514a8da95020107f6e563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.956094 10568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86682656c9514a8da95020107f6e563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.956095 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86682656c9514a8da95020107f6e563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:00.961297 10935 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: Bootstrap starting.
I20260928 05:23:00.962831 10936 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: Bootstrap starting.
I20260928 05:23:00.962846 10935 tablet_bootstrap.cc:654] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: Neither blocks nor log segments found. Creating new log.
W20260928 05:23:00.963251 10917 tablet.cc:2369] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:00.964363 10936 tablet_bootstrap.cc:654] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:00.965106 10935 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: No bootstrap required, opened a new log
I20260928 05:23:00.965229 10935 ts_tablet_manager.cc:1397] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:23:00.965318 10937 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: Bootstrap starting.
I20260928 05:23:00.965967 10935 raft_consensus.cc:348] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.966216 10935 raft_consensus.cc:374] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.966349 10935 raft_consensus.cc:729] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.966688 10936 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: No bootstrap required, opened a new log
I20260928 05:23:00.966566 10935 consensus_queue.cc:260] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.966799 10937 tablet_bootstrap.cc:654] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:00.966991 10942 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:00.967173 10935 ts_tablet_manager.cc:1428] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:00.967273 10942 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:00.967502 10936 ts_tablet_manager.cc:1397] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:23:00.968223 10936 raft_consensus.cc:348] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.968447 10936 raft_consensus.cc:374] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.968508 10936 raft_consensus.cc:729] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.968683 10936 consensus_queue.cc:260] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.969169 10936 ts_tablet_manager.cc:1428] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:23:00.969040 10941 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:00.969394 10941 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 05:23:00.975311 10778 tablet.cc:2369] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:00.976542 10937 tablet_bootstrap.cc:492] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: No bootstrap required, opened a new log
I20260928 05:23:00.976672 10937 ts_tablet_manager.cc:1397] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260928 05:23:00.977301 10937 raft_consensus.cc:348] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.977497 10937 raft_consensus.cc:374] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:00.977561 10937 raft_consensus.cc:729] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c626403d460c4018b8913140cfe0b351, State: Initialized, Role: FOLLOWER
I20260928 05:23:00.977798 10937 consensus_queue.cc:260] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:00.978360 10937 ts_tablet_manager.cc:1428] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:23:00.978305 10943 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:00.978739 10943 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 05:23:01.002956 10639 tablet.cc:2369] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:01.117568 10941 raft_consensus.cc:482] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:01.117856 10941 raft_consensus.cc:504] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.119652 10941 leader_election.cc:288] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:01.127729 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.120277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:43337 (local address 127.8.124.193:33913)
0928 05:23:01.120447 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.120453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.121386 (+ 933us) server_negotiation.cc:407] Connection header received
0928 05:23:01.121570 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.121575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.121669 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.121719 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:01.122434 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.123517 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.125780 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.126601 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.126740 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.126812 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.127331 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.127456 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:23:01.127753 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:59917 (local address 127.8.124.195:38463)
0928 05:23:01.120363 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.120370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.121007 (+ 637us) server_negotiation.cc:407] Connection header received
0928 05:23:01.121323 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.121328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.121405 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.121490 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:01.122059 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.122992 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.125340 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.126615 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.126769 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.126846 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.127374 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.127507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:23:01.127764 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33913 (local address 127.8.124.194:43337)
0928 05:23:01.121273 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.121297 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.121416 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.121873 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.121877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.121894 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:01.122260 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.122271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.123656 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.123661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.125593 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.125628 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.127031 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.127036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.127161 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.127211 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.127288 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.127407 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":553,"thread_start_us":564,"threads_started":1}
I20260928 05:23:01.128355 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86682656c9514a8da95020107f6e563f" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351" is_pre_election: true
I20260928 05:23:01.128613 10588 raft_consensus.cc:2454] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:01.128662 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38463 (local address 127.8.124.194:59917)
0928 05:23:01.120904 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.120922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.121128 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.121511 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.121516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.121540 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:01.121892 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.121904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.123138 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.123143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.125185 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.125199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.127008 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.127018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.127213 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.127245 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.128215 (+ 970us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.128335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":4,"thread_start_us":713,"threads_started":1}
I20260928 05:23:01.129153 10659 leader_election.cc:302] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:01.129570 10941 raft_consensus.cc:2781] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:01.129731 10941 raft_consensus.cc:482] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:01.129532 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86682656c9514a8da95020107f6e563f" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09" is_pre_election: true
I20260928 05:23:01.129824 10941 raft_consensus.cc:3037] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.129972 10870 raft_consensus.cc:2454] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:01.132797 10941 raft_consensus.cc:504] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.133441 10941 leader_election.cc:288] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Requested vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:01.134035 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86682656c9514a8da95020107f6e563f" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351"
I20260928 05:23:01.134276 10588 raft_consensus.cc:3037] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.134227 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86682656c9514a8da95020107f6e563f" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09"
I20260928 05:23:01.134418 10870 raft_consensus.cc:3037] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.137164 10588 raft_consensus.cc:2454] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:01.137188 10870 raft_consensus.cc:2454] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:01.137631 10659 leader_election.cc:302] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, ac1a597683b745ecab8c4d13bbe88f09, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:01.137987 10941 raft_consensus.cc:2781] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:01.138422 10941 raft_consensus.cc:686] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [term 1 LEADER]: Becoming Leader. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Running, Role: LEADER
I20260928 05:23:01.138971 10941 consensus_queue.cc:237] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.142069 10958 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: TxnStatusTablet state changed. Reason: New leader 09049f079f2e4189918a77b3ddf2bc0e. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.142376 10958 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:01.143045 10960 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:23:01.143306 10960 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:23:01.144012 10430 catalog_manager.cc:5495] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194). New cstate: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:01.163160 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.157908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.0.0.1:44836)
0928 05:23:01.158087 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.158109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.158244 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.158723 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.158727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.158739 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:01.158900 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.158908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.159896 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.159901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.160872 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.160883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.161564 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.161570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.161816 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.161894 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.162650 (+ 756us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.162670 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.162750 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.162890 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:23:01.163175 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.157780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.8.124.194:43967)
0928 05:23:01.157920 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.157924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.158166 (+ 242us) server_negotiation.cc:407] Connection header received
0928 05:23:01.158450 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.158454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.158564 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.158673 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.159025 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.159783 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.160997 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.161423 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.161529 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.162060 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.162552 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.162674 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.162768 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.162839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:23:01.171272 10588 raft_consensus.cc:1261] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.172328 10941 consensus_queue.cc:1035] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:01.172492 10870 raft_consensus.cc:1261] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.173431 10941 consensus_queue.cc:1035] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:01.193042 10963 mvcc.cc:205] Tried to move back new op lower bound from 7334186930864754688 to 7334186930748887040. Current Snapshot: MvccSnapshot[applied={T|T < 7334186930864754688}]
I20260928 05:23:01.193943 10943 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: TxnStatusTablet state changed. Reason: New leader 09049f079f2e4189918a77b3ddf2bc0e. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.194240 10943 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:01.195686 10958 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.195955 10958 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:01.198367 10942 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: TxnStatusTablet state changed. Reason: New leader 09049f079f2e4189918a77b3ddf2bc0e. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.198663 10942 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:01.201077 10958 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.201330 10958 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P 09049f079f2e4189918a77b3ddf2bc0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:01.201922 10943 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.202128 10943 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P c626403d460c4018b8913140cfe0b351: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:01.203336 10966 mvcc.cc:205] Tried to move back new op lower bound from 7334186930864754688 to 7334186930748887040. Current Snapshot: MvccSnapshot[applied={T|T < 7334186930864754688}]
I20260928 05:23:01.210191 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.204875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.8.124.195:38463)
0928 05:23:01.205034 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.205040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.205503 (+ 463us) server_negotiation.cc:407] Connection header received
0928 05:23:01.205582 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.205586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.205706 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.205826 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.206161 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.206863 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.208143 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.208539 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.208641 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.209051 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.209548 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.209711 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.209798 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.209881 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:23:01.213282 10942 tablet_replica.cc:439] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } } }
I20260928 05:23:01.213567 10942 tablet_replica.cc:442] T 86682656c9514a8da95020107f6e563f P ac1a597683b745ecab8c4d13bbe88f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:01.227927 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.220698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:33287 (local address 127.8.124.194:43967)
0928 05:23:01.221003 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.221009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.221344 (+ 335us) server_negotiation.cc:407] Connection header received
0928 05:23:01.221497 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.221502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.221586 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.221712 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.222350 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.223503 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.225386 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.226109 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.226229 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.227196 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.227329 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.227416 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.227557 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.227635 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260928 05:23:01.228013 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.220627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.8.124.195:33287)
0928 05:23:01.221256 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.221271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.221384 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.221729 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.221734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.221747 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:01.222137 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.222149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.223641 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.223646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.225241 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.225256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.226755 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.226764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.226923 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.226987 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.227435 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.227446 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.227524 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.227659 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":306,"threads_started":1}
I20260928 05:23:01.237375 10587 raft_consensus.cc:1261] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Refusing update from remote peer ac1a597683b745ecab8c4d13bbe88f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.237751 10732 raft_consensus.cc:1261] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Refusing update from remote peer ac1a597683b745ecab8c4d13bbe88f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.238155 10945 consensus_queue.cc:1035] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:01.238763 10945 consensus_queue.cc:1035] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:01.270339 10964 mvcc.cc:205] Tried to move back new op lower bound from 7334186931142541312 to 7334186929794748416. Current Snapshot: MvccSnapshot[applied={T|T < 7334186931142541312}]
I20260928 05:23:01.300064 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.285453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:55379 (local address 127.8.124.195:38463)
0928 05:23:01.285659 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.285665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.294222 (+ 8557us) server_negotiation.cc:407] Connection header received
0928 05:23:01.294426 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.294432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.294519 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.294610 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.295246 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.296381 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.298005 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.298684 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.298824 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.299319 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.299445 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.299523 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.299685 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.299775 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:01.300168 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.285372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38463 (local address 127.8.124.194:55379)
0928 05:23:01.294111 (+ 8739us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.294128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.294283 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.294638 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.294642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.294654 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:01.295094 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.295109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.296527 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.296532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.297842 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.297876 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.298885 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.298892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.299074 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.299139 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.299570 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.299581 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.299655 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.299798 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8550,"thread_start_us":8418,"threads_started":1}
I20260928 05:23:01.337935 10429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:01.338229 10429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:01.342097 10429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Sending DeleteTablet for 3 replicas of tablet e48b568b954840ab8a3ff4a5d6dbb80d
I20260928 05:23:01.343232 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:01 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:34246
I20260928 05:23:01.343230 10707 tablet_service.cc:1514] Processing DeleteTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:01 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44834
I20260928 05:23:01.344647 10568 tablet_service.cc:1514] Processing DeleteTablet for tablet e48b568b954840ab8a3ff4a5d6dbb80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:01 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:43166
I20260928 05:23:01.352044 10429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:01.352717 10429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:01.353404 10986 tablet_replica.cc:330] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: stopping tablet replica
I20260928 05:23:01.354441 10986 raft_consensus.cc:2229] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:01.361312 10706 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 (DEFAULT_TABLE table=test-table [id=6285be4ea61c463aa6e1e19ae3d49081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:01.361846 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc06359deb4dcb815c887ed9f6eb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:01.362288 10986 raft_consensus.cc:2258] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:01.364270 10984 tablet_replica.cc:330] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: stopping tablet replica
I20260928 05:23:01.364838 10984 raft_consensus.cc:2229] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:01.365223 10984 raft_consensus.cc:2258] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:01.366888 10845 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 (DEFAULT_TABLE table=test-table [id=6285be4ea61c463aa6e1e19ae3d49081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:01.367110 10984 ts_tablet_manager.cc:1905] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:01.367447 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc06359deb4dcb815c887ed9f6eb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:01.374938 10984 ts_tablet_manager.cc:1918] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:01.375216 10984 log.cc:1196] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/e48b568b954840ab8a3ff4a5d6dbb80d
I20260928 05:23:01.375219 10568 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 (DEFAULT_TABLE table=test-table [id=6285be4ea61c463aa6e1e19ae3d49081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:01.375705 10568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc06359deb4dcb815c887ed9f6eb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:01.376006 10984 ts_tablet_manager.cc:1939] T e48b568b954840ab8a3ff4a5d6dbb80d P 09049f079f2e4189918a77b3ddf2bc0e: Deleting consensus metadata
I20260928 05:23:01.378228 10418 catalog_manager.cc:4841] TS 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967): tablet e48b568b954840ab8a3ff4a5d6dbb80d (table test-table [id=addee9da48a4447b801da9769c9ae644]) successfully deleted
I20260928 05:23:01.382793 10936 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Bootstrap starting.
I20260928 05:23:01.384460 10936 tablet_bootstrap.cc:654] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:01.386823 10986 ts_tablet_manager.cc:1905] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:01.390326 10935 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Bootstrap starting.
I20260928 05:23:01.391978 10935 tablet_bootstrap.cc:654] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:01.390312 10937 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Bootstrap starting.
I20260928 05:23:01.397323 10937 tablet_bootstrap.cc:654] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:01.398173 10936 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: No bootstrap required, opened a new log
I20260928 05:23:01.398346 10936 ts_tablet_manager.cc:1397] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260928 05:23:01.399098 10936 raft_consensus.cc:348] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.399340 10936 raft_consensus.cc:374] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:01.399427 10936 raft_consensus.cc:729] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Initialized, Role: FOLLOWER
I20260928 05:23:01.399685 10936 consensus_queue.cc:260] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.401090 10936 ts_tablet_manager.cc:1428] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 05:23:01.402505 10935 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: No bootstrap required, opened a new log
I20260928 05:23:01.402669 10935 ts_tablet_manager.cc:1397] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 05:23:01.403405 10935 raft_consensus.cc:348] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.403636 10935 raft_consensus.cc:374] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:01.403712 10935 raft_consensus.cc:729] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Initialized, Role: FOLLOWER
I20260928 05:23:01.403977 10935 consensus_queue.cc:260] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.406482 10935 ts_tablet_manager.cc:1428] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:23:01.408532 10985 tablet_replica.cc:330] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: stopping tablet replica
I20260928 05:23:01.409071 10985 raft_consensus.cc:2229] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:01.409426 10986 ts_tablet_manager.cc:1918] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:01.409696 10937 tablet_bootstrap.cc:492] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: No bootstrap required, opened a new log
I20260928 05:23:01.409919 10986 log.cc:1196] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/e48b568b954840ab8a3ff4a5d6dbb80d
I20260928 05:23:01.410583 10985 raft_consensus.cc:2258] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:01.411365 10986 ts_tablet_manager.cc:1939] T e48b568b954840ab8a3ff4a5d6dbb80d P c626403d460c4018b8913140cfe0b351: Deleting consensus metadata
I20260928 05:23:01.409940 10937 ts_tablet_manager.cc:1397] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20260928 05:23:01.413143 10416 catalog_manager.cc:4841] TS c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913): tablet e48b568b954840ab8a3ff4a5d6dbb80d (table test-table [id=addee9da48a4447b801da9769c9ae644]) successfully deleted
I20260928 05:23:01.413656 10937 raft_consensus.cc:348] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.413897 10937 raft_consensus.cc:374] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:01.413972 10937 raft_consensus.cc:729] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c626403d460c4018b8913140cfe0b351, State: Initialized, Role: FOLLOWER
I20260928 05:23:01.414235 10937 consensus_queue.cc:260] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.414897 10937 ts_tablet_manager.cc:1428] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:01.415301 10985 ts_tablet_manager.cc:1905] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:01.422618 10985 ts_tablet_manager.cc:1918] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:01.422788 10985 log.cc:1196] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/e48b568b954840ab8a3ff4a5d6dbb80d
I20260928 05:23:01.423234 10985 ts_tablet_manager.cc:1939] T e48b568b954840ab8a3ff4a5d6dbb80d P ac1a597683b745ecab8c4d13bbe88f09: Deleting consensus metadata
I20260928 05:23:01.424937 10417 catalog_manager.cc:4841] TS ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463): tablet e48b568b954840ab8a3ff4a5d6dbb80d (table test-table [id=addee9da48a4447b801da9769c9ae644]) successfully deleted
I20260928 05:23:01.500789 10943 raft_consensus.cc:482] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:01.501009 10943 raft_consensus.cc:504] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.502723 10943 leader_election.cc:288] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:01.507010 10958 raft_consensus.cc:482] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:01.507195 10958 raft_consensus.cc:504] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.507912 10958 leader_election.cc:288] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:01.508708 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09" is_pre_election: true
I20260928 05:23:01.508965 10870 raft_consensus.cc:2454] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:01.509473 10660 leader_election.cc:302] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, ac1a597683b745ecab8c4d13bbe88f09; no voters:
I20260928 05:23:01.509989 10958 raft_consensus.cc:2781] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:01.510114 10958 raft_consensus.cc:482] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:01.510222 10958 raft_consensus.cc:3037] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.513324 10958 raft_consensus.cc:504] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.515339 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09"
I20260928 05:23:01.515566 10870 raft_consensus.cc:3037] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.518853 10870 raft_consensus.cc:2454] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:01.519357 10660 leader_election.cc:302] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, ac1a597683b745ecab8c4d13bbe88f09; no voters:
I20260928 05:23:01.519631 10941 raft_consensus.cc:2781] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:01.520035 10941 raft_consensus.cc:686] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 LEADER]: Becoming Leader. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Running, Role: LEADER
I20260928 05:23:01.520325 10941 consensus_queue.cc:237] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:01.522063 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351" is_pre_election: true
I20260928 05:23:01.522312 10588 raft_consensus.cc:2454] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:01.522964 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351"
I20260928 05:23:01.523183 10588 raft_consensus.cc:3037] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:01.526229 10588 raft_consensus.cc:2454] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:01.528083 10958 leader_election.cc:288] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Requested vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:01.532290 10430 catalog_manager.cc:5495] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194). New cstate: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:01.542315 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.503339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:38677 (local address 127.8.124.194:43967)
0928 05:23:01.503552 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.503558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.513872 (+ 10314us) server_negotiation.cc:407] Connection header received
0928 05:23:01.514072 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.514078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.514153 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.514277 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:01.535067 (+ 20790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.536227 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.539594 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.540964 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.541105 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.541181 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.541862 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.542006 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:23:01.543071 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.502988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.8.124.193:38677)
0928 05:23:01.513765 (+ 10777us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.513792 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.513952 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.534400 (+ 20448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.534406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.534443 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:01.534842 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.534858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.536443 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.536449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.539396 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.539411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.541421 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.541428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.541664 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.541699 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.542557 (+ 858us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.542693 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10638,"thread_start_us":18498,"threads_started":1}
I20260928 05:23:01.543766 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09049f079f2e4189918a77b3ddf2bc0e" is_pre_election: true
I20260928 05:23:01.546124 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.503499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:47741 (local address 127.8.124.195:38463)
0928 05:23:01.503678 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.503684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.538777 (+ 35093us) server_negotiation.cc:407] Connection header received
0928 05:23:01.539075 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.539081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.539163 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.539226 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:01.540064 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.541203 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.543890 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.545161 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.545330 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.545400 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.545719 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.545848 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:23:01.546154 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.503145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38463 (local address 127.8.124.193:47741)
0928 05:23:01.538653 (+ 35508us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.538680 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.538896 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.539511 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.539516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.539543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:01.539880 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.539892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.541408 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.541413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.543724 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.543738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.545365 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.545372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.545535 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.545568 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.545692 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.545827 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35371,"thread_start_us":35237,"threads_started":1}
I20260928 05:23:01.546907 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc06359deb4dcb815c887ed9f6eb49" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09" is_pre_election: true
I20260928 05:23:01.547237 10870 raft_consensus.cc:2379] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c626403d460c4018b8913140cfe0b351 in current term 1: Already voted for candidate 09049f079f2e4189918a77b3ddf2bc0e in this term.
I20260928 05:23:01.547699 10520 leader_election.cc:302] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c626403d460c4018b8913140cfe0b351; no voters: 09049f079f2e4189918a77b3ddf2bc0e, ac1a597683b745ecab8c4d13bbe88f09
I20260928 05:23:01.548110 10943 raft_consensus.cc:2726] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:23:01.592885 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.586646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.8.124.194:43967)
0928 05:23:01.586815 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.586821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.587093 (+ 272us) server_negotiation.cc:407] Connection header received
0928 05:23:01.587267 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.587272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.587349 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.587435 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.587834 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.588938 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.590875 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.591583 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.591718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.592172 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.592307 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.592403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.592539 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.592614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:01.608129 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.602232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:59403 (local address 127.8.124.194:43967)
0928 05:23:01.602417 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:01.602422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:01.602641 (+ 219us) server_negotiation.cc:407] Connection header received
0928 05:23:01.602805 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:01.602810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:01.602883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:01.602967 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:01.603314 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.604390 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.606270 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.606839 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.606949 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.607375 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.607516 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.607597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:01.607724 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:23:01.607807 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:23:01.608439 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:01.602244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.8.124.194:59403)
0928 05:23:01.602557 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:01.602576 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:01.602694 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:01.602991 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:01.602996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:01.603007 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:01.603191 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.603201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.604520 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.604526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.606126 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:01.606141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:01.606995 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:01.607001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:01.607176 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:01.607224 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:01.607614 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:01.607625 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:23:01.607702 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:23:01.607854 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":3}
I20260928 05:23:01.617508 10870 raft_consensus.cc:1261] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.617511 10588 raft_consensus.cc:1261] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:01.618106 10992 consensus_queue.cc:1035] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:01.618520 10958 consensus_queue.cc:1035] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:23:01.641053 10963 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:23:03.233916 10430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:03.234179 10430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:03.238284 10430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Sending DeleteTablet for 3 replicas of tablet 95fc06359deb4dcb815c887ed9f6eb49
I20260928 05:23:03.238993 10706 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44834
I20260928 05:23:03.239018 10568 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:43166
I20260928 05:23:03.239598 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fc06359deb4dcb815c887ed9f6eb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:34246
I20260928 05:23:03.240231 11019 tablet_replica.cc:330] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: stopping tablet replica
I20260928 05:23:03.240425 11019 raft_consensus.cc:2229] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:03.240933 11019 raft_consensus.cc:2258] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.241894 11019 ts_tablet_manager.cc:1905] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.245940 11020 tablet_replica.cc:330] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: stopping tablet replica
I20260928 05:23:03.246110 11018 tablet_replica.cc:330] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: stopping tablet replica
I20260928 05:23:03.246116 11020 raft_consensus.cc:2229] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:03.246284 11018 raft_consensus.cc:2229] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:03.246322 11020 raft_consensus.cc:2258] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.246475 11018 raft_consensus.cc:2258] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.247287 11018 ts_tablet_manager.cc:1905] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.249539 11019 ts_tablet_manager.cc:1918] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:03.249708 11019 log.cc:1196] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/95fc06359deb4dcb815c887ed9f6eb49
I20260928 05:23:03.249744 11020 ts_tablet_manager.cc:1905] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.250128 11019 ts_tablet_manager.cc:1939] T 95fc06359deb4dcb815c887ed9f6eb49 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting consensus metadata
I20260928 05:23:03.251183 10418 catalog_manager.cc:4841] TS 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967): tablet 95fc06359deb4dcb815c887ed9f6eb49 (table test-table [id=6285be4ea61c463aa6e1e19ae3d49081]) successfully deleted
I20260928 05:23:03.253563 11018 ts_tablet_manager.cc:1918] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:03.253733 11018 log.cc:1196] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/95fc06359deb4dcb815c887ed9f6eb49
I20260928 05:23:03.254110 11018 ts_tablet_manager.cc:1939] T 95fc06359deb4dcb815c887ed9f6eb49 P c626403d460c4018b8913140cfe0b351: Deleting consensus metadata
I20260928 05:23:03.255189 10416 catalog_manager.cc:4841] TS c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913): tablet 95fc06359deb4dcb815c887ed9f6eb49 (table test-table [id=6285be4ea61c463aa6e1e19ae3d49081]) successfully deleted
I20260928 05:23:03.255905 11020 ts_tablet_manager.cc:1918] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:03.256021 11020 log.cc:1196] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/95fc06359deb4dcb815c887ed9f6eb49
I20260928 05:23:03.256364 11020 ts_tablet_manager.cc:1939] T 95fc06359deb4dcb815c887ed9f6eb49 P ac1a597683b745ecab8c4d13bbe88f09: Deleting consensus metadata
I20260928 05:23:03.257256 10417 catalog_manager.cc:4841] TS ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463): tablet 95fc06359deb4dcb815c887ed9f6eb49 (table test-table [id=6285be4ea61c463aa6e1e19ae3d49081]) successfully deleted
Sep 28 05:23:03 dist-test-slave-515x krb5kdc[10386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:23:03 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:03.292364 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.274029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.8.124.254:36481)
0928 05:23:03.274489 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:03.274493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:03.274544 (+ 51us) server_negotiation.cc:407] Connection header received
0928 05:23:03.274713 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:03.274717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:03.274779 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:03.274868 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:03.276041 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.276841 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.278075 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.278607 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.278706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.287957 (+ 9251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:03.287962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:03.287966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:03.288003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:03.290342 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:03.290931 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:03.290935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:03.290937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:03.291046 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:03.291305 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:03.291308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:03.291309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:03.291561 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:03.291730 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:03.291947 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 05:23:03.292103 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":248,"threads_started":1}
I20260928 05:23:03.295879 10430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:03.296447 10430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:03.302670 10568 tablet_service.cc:1467] Processing CreateTablet for tablet 376da45e89be4f66af4dee93a382fcb1 (DEFAULT_TABLE table=test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.302755 10706 tablet_service.cc:1467] Processing CreateTablet for tablet 376da45e89be4f66af4dee93a382fcb1 (DEFAULT_TABLE table=test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.302999 10846 tablet_service.cc:1467] Processing CreateTablet for tablet 376da45e89be4f66af4dee93a382fcb1 (DEFAULT_TABLE table=test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.303104 10568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376da45e89be4f66af4dee93a382fcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.303122 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376da45e89be4f66af4dee93a382fcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.303310 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376da45e89be4f66af4dee93a382fcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.308818 11029 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Bootstrap starting.
I20260928 05:23:03.309444 11030 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Bootstrap starting.
I20260928 05:23:03.310153 11031 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Bootstrap starting.
I20260928 05:23:03.310431 11029 tablet_bootstrap.cc:654] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.311120 11030 tablet_bootstrap.cc:654] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.311575 11031 tablet_bootstrap.cc:654] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.313014 11029 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: No bootstrap required, opened a new log
I20260928 05:23:03.313113 11030 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: No bootstrap required, opened a new log
I20260928 05:23:03.313136 11029 ts_tablet_manager.cc:1397] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:23:03.313254 11030 ts_tablet_manager.cc:1397] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 05:23:03.313836 11029 raft_consensus.cc:348] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.314011 11029 raft_consensus.cc:374] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.313907 11030 raft_consensus.cc:348] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.314095 11029 raft_consensus.cc:729] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.314116 11030 raft_consensus.cc:374] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.314194 11031 tablet_bootstrap.cc:492] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: No bootstrap required, opened a new log
I20260928 05:23:03.314220 11030 raft_consensus.cc:729] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.314359 11031 ts_tablet_manager.cc:1397] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 05:23:03.314316 11029 consensus_queue.cc:260] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.314441 11030 consensus_queue.cc:260] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.314949 11030 ts_tablet_manager.cc:1428] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 05:23:03.315035 11029 ts_tablet_manager.cc:1428] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:03.315598 11031 raft_consensus.cc:348] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.315883 11031 raft_consensus.cc:374] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.316023 11031 raft_consensus.cc:729] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c626403d460c4018b8913140cfe0b351, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.316256 11031 consensus_queue.cc:260] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.317162 11031 ts_tablet_manager.cc:1428] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:03.323730 11013 raft_consensus.cc:482] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:03.323899 11013 raft_consensus.cc:504] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.324488 11013 leader_election.cc:288] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:03.324867 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376da45e89be4f66af4dee93a382fcb1" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351" is_pre_election: true
I20260928 05:23:03.324977 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376da45e89be4f66af4dee93a382fcb1" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09" is_pre_election: true
I20260928 05:23:03.325084 10588 raft_consensus.cc:2454] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:03.325198 10870 raft_consensus.cc:2454] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 0.
I20260928 05:23:03.325513 10659 leader_election.cc:302] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:03.325819 11013 raft_consensus.cc:2781] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:03.325922 11013 raft_consensus.cc:482] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:03.326007 11013 raft_consensus.cc:3037] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.327991 11013 raft_consensus.cc:504] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.328428 11013 leader_election.cc:288] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Requested vote from peers c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:03.328840 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376da45e89be4f66af4dee93a382fcb1" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c626403d460c4018b8913140cfe0b351"
I20260928 05:23:03.328851 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376da45e89be4f66af4dee93a382fcb1" candidate_uuid: "09049f079f2e4189918a77b3ddf2bc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09"
I20260928 05:23:03.329031 10870 raft_consensus.cc:3037] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.329036 10588 raft_consensus.cc:3037] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.331054 10870 raft_consensus.cc:2454] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:03.331164 10588 raft_consensus.cc:2454] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09049f079f2e4189918a77b3ddf2bc0e in term 1.
I20260928 05:23:03.331514 10659 leader_election.cc:302] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09049f079f2e4189918a77b3ddf2bc0e, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:03.331800 11013 raft_consensus.cc:2781] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:03.331918 11013 raft_consensus.cc:686] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 LEADER]: Becoming Leader. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Running, Role: LEADER
I20260928 05:23:03.332170 11013 consensus_queue.cc:237] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.334611 10431 catalog_manager.cc:5495] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194). New cstate: current_term: 1 leader_uuid: "09049f079f2e4189918a77b3ddf2bc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:03.357829 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.353411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.8.124.194:43967)
0928 05:23:03.353855 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:03.353859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:03.353876 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:03.353920 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:03.353923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:03.353979 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:03.354055 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:03.354340 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.355068 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.356311 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.356751 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.356860 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.357211 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.357315 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.357376 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:03.357455 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 05:23:03.357509 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
I20260928 05:23:03.365754 10588 raft_consensus.cc:1261] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:03.365741 10870 raft_consensus.cc:1261] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Refusing update from remote peer 09049f079f2e4189918a77b3ddf2bc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:03.366235 11000 consensus_queue.cc:1035] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:03.366516 11038 consensus_queue.cc:1035] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:03.404961 10431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:03.405153 10431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:03.407749 10431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Sending DeleteTablet for 3 replicas of tablet 376da45e89be4f66af4dee93a382fcb1
I20260928 05:23:03.408449 10568 tablet_service.cc:1514] Processing DeleteTablet for tablet 376da45e89be4f66af4dee93a382fcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:43166
I20260928 05:23:03.408612 10706 tablet_service.cc:1514] Processing DeleteTablet for tablet 376da45e89be4f66af4dee93a382fcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44834
I20260928 05:23:03.408648 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet 376da45e89be4f66af4dee93a382fcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:34246
I20260928 05:23:03.408715 11018 tablet_replica.cc:330] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: stopping tablet replica
I20260928 05:23:03.408885 11020 tablet_replica.cc:330] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: stopping tablet replica
I20260928 05:23:03.408885 11018 raft_consensus.cc:2229] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:03.408919 11019 tablet_replica.cc:330] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: stopping tablet replica
I20260928 05:23:03.409049 11018 raft_consensus.cc:2258] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.409058 11020 raft_consensus.cc:2229] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:03.409086 11019 raft_consensus.cc:2229] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:03.409189 11020 raft_consensus.cc:2258] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.409364 11019 raft_consensus.cc:2258] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:03.410270 11020 ts_tablet_manager.cc:1905] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.410374 11019 ts_tablet_manager.cc:1905] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.410441 11018 ts_tablet_manager.cc:1905] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:03.414790 11020 ts_tablet_manager.cc:1918] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:03.414930 11020 log.cc:1196] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/376da45e89be4f66af4dee93a382fcb1
I20260928 05:23:03.415352 11020 ts_tablet_manager.cc:1939] T 376da45e89be4f66af4dee93a382fcb1 P ac1a597683b745ecab8c4d13bbe88f09: Deleting consensus metadata
I20260928 05:23:03.415493 10431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:03.416188 10431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:03.416311 10417 catalog_manager.cc:4841] TS ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463): tablet 376da45e89be4f66af4dee93a382fcb1 (table test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]) successfully deleted
I20260928 05:23:03.416553 11018 ts_tablet_manager.cc:1918] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:03.416702 11018 log.cc:1196] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/376da45e89be4f66af4dee93a382fcb1
I20260928 05:23:03.417078 11018 ts_tablet_manager.cc:1939] T 376da45e89be4f66af4dee93a382fcb1 P c626403d460c4018b8913140cfe0b351: Deleting consensus metadata
I20260928 05:23:03.418118 10416 catalog_manager.cc:4841] TS c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913): tablet 376da45e89be4f66af4dee93a382fcb1 (table test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]) successfully deleted
I20260928 05:23:03.418785 11019 ts_tablet_manager.cc:1918] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:03.418936 11019 log.cc:1196] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/376da45e89be4f66af4dee93a382fcb1
I20260928 05:23:03.419312 11019 ts_tablet_manager.cc:1939] T 376da45e89be4f66af4dee93a382fcb1 P 09049f079f2e4189918a77b3ddf2bc0e: Deleting consensus metadata
I20260928 05:23:03.420255 10418 catalog_manager.cc:4841] TS 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967): tablet 376da45e89be4f66af4dee93a382fcb1 (table test-table [id=46e21ddd5a6c4c8795847018d7dd2f8d]) successfully deleted
I20260928 05:23:03.422807 10568 tablet_service.cc:1467] Processing CreateTablet for tablet a13b7d7784484f9e9c87ba129461aedb (DEFAULT_TABLE table=test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.423015 10706 tablet_service.cc:1467] Processing CreateTablet for tablet a13b7d7784484f9e9c87ba129461aedb (DEFAULT_TABLE table=test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.423215 10568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13b7d7784484f9e9c87ba129461aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.423161 10846 tablet_service.cc:1467] Processing CreateTablet for tablet a13b7d7784484f9e9c87ba129461aedb (DEFAULT_TABLE table=test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:03.423381 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13b7d7784484f9e9c87ba129461aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.423499 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13b7d7784484f9e9c87ba129461aedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:03.427347 11031 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Bootstrap starting.
I20260928 05:23:03.428506 11031 tablet_bootstrap.cc:654] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.429230 11030 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Bootstrap starting.
I20260928 05:23:03.429471 11029 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Bootstrap starting.
I20260928 05:23:03.430747 11030 tablet_bootstrap.cc:654] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.430850 11029 tablet_bootstrap.cc:654] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:03.430855 11031 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: No bootstrap required, opened a new log
I20260928 05:23:03.430989 11031 ts_tablet_manager.cc:1397] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:23:03.431687 11031 raft_consensus.cc:348] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.431941 11031 raft_consensus.cc:374] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.432009 11031 raft_consensus.cc:729] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c626403d460c4018b8913140cfe0b351, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.432292 11031 consensus_queue.cc:260] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.432523 11029 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: No bootstrap required, opened a new log
I20260928 05:23:03.432662 11029 ts_tablet_manager.cc:1397] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:03.432814 11031 ts_tablet_manager.cc:1428] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:03.433974 11030 tablet_bootstrap.cc:492] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: No bootstrap required, opened a new log
I20260928 05:23:03.434129 11030 ts_tablet_manager.cc:1397] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 05:23:03.434618 11030 raft_consensus.cc:348] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.434653 11029 raft_consensus.cc:348] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.434820 11030 raft_consensus.cc:374] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.434894 11030 raft_consensus.cc:729] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09049f079f2e4189918a77b3ddf2bc0e, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.434894 11029 raft_consensus.cc:374] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:03.434965 11029 raft_consensus.cc:729] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a597683b745ecab8c4d13bbe88f09, State: Initialized, Role: FOLLOWER
I20260928 05:23:03.435113 11030 consensus_queue.cc:260] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.435180 11029 consensus_queue.cc:260] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.435662 11029 ts_tablet_manager.cc:1428] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 05:23:03.435703 11030 ts_tablet_manager.cc:1428] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 05:23:03.436472 11036 raft_consensus.cc:482] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:03.436661 11036 raft_consensus.cc:504] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.437286 11036 leader_election.cc:288] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:03.437953 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13b7d7784484f9e9c87ba129461aedb" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09049f079f2e4189918a77b3ddf2bc0e" is_pre_election: true
I20260928 05:23:03.438002 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13b7d7784484f9e9c87ba129461aedb" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09" is_pre_election: true
I20260928 05:23:03.438189 10870 raft_consensus.cc:2454] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c626403d460c4018b8913140cfe0b351 in term 0.
I20260928 05:23:03.438162 10732 raft_consensus.cc:2454] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c626403d460c4018b8913140cfe0b351 in term 0.
I20260928 05:23:03.438560 10520 leader_election.cc:302] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac1a597683b745ecab8c4d13bbe88f09, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:03.439065 11036 raft_consensus.cc:2781] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:03.439157 11036 raft_consensus.cc:482] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:03.439255 11036 raft_consensus.cc:3037] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.441069 11036 raft_consensus.cc:504] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.441697 11036 leader_election.cc:288] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 election: Requested vote from peers 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967), ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463)
I20260928 05:23:03.442234 10870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13b7d7784484f9e9c87ba129461aedb" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a597683b745ecab8c4d13bbe88f09"
I20260928 05:23:03.442385 10870 raft_consensus.cc:3037] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.442342 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13b7d7784484f9e9c87ba129461aedb" candidate_uuid: "c626403d460c4018b8913140cfe0b351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09049f079f2e4189918a77b3ddf2bc0e"
I20260928 05:23:03.442518 10732 raft_consensus.cc:3037] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:03.444141 10870 raft_consensus.cc:2454] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c626403d460c4018b8913140cfe0b351 in term 1.
I20260928 05:23:03.444272 10732 raft_consensus.cc:2454] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c626403d460c4018b8913140cfe0b351 in term 1.
I20260928 05:23:03.444450 10520 leader_election.cc:302] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac1a597683b745ecab8c4d13bbe88f09, c626403d460c4018b8913140cfe0b351; no voters:
I20260928 05:23:03.444715 11036 raft_consensus.cc:2781] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:03.445128 11036 raft_consensus.cc:686] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 1 LEADER]: Becoming Leader. State: Replica: c626403d460c4018b8913140cfe0b351, State: Running, Role: LEADER
I20260928 05:23:03.445495 11036 consensus_queue.cc:237] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } }
I20260928 05:23:03.448482 10431 catalog_manager.cc:5495] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 reported cstate change: term changed from 0 to 1, leader changed from <none> to c626403d460c4018b8913140cfe0b351 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "c626403d460c4018b8913140cfe0b351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c626403d460c4018b8913140cfe0b351" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:03.468991 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.463515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.8.124.193:33913)
0928 05:23:03.464024 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:03.464028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:03.464048 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:23:03.464125 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:03.464129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:03.464196 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:03.464406 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:03.464797 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.465553 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.466830 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.467247 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.467352 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.467819 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.468413 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.468547 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:03.468629 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 05:23:03.468710 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":308,"threads_started":1}
I20260928 05:23:03.483580 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.477878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:38493 (local address 127.8.124.194:43967)
0928 05:23:03.478042 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:03.478048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:03.478637 (+ 589us) server_negotiation.cc:407] Connection header received
0928 05:23:03.478841 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:03.478844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:03.478906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:03.478967 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:03.479392 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.480162 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.482030 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.482475 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.482574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.483028 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.483134 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.483204 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:03.483313 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 05:23:03.483376 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:23:03.483739 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.477929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43967 (local address 127.8.124.193:38493)
0928 05:23:03.478558 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:03.478572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:03.478706 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:03.478992 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:03.478995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:03.479006 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:03.479265 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:03.479274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.480348 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.480353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:03.481884 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:03.481898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.482636 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.482641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:03.482839 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.482895 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.483226 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.483234 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:23:03.483297 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 05:23:03.483395 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":325,"threads_started":1}
I20260928 05:23:03.491210 10870 raft_consensus.cc:1261] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Refusing update from remote peer c626403d460c4018b8913140cfe0b351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:03.491343 10732 raft_consensus.cc:1261] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Refusing update from remote peer c626403d460c4018b8913140cfe0b351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:03.491724 11047 consensus_queue.cc:1035] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a597683b745ecab8c4d13bbe88f09" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:03.492116 11036 consensus_queue.cc:1035] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09049f079f2e4189918a77b3ddf2bc0e" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:03.515715 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.510633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57667 (local address 127.8.124.193:33913)
0928 05:23:03.511029 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:03.511035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:03.511386 (+ 351us) server_negotiation.cc:407] Connection header received
0928 05:23:03.511509 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:03.511513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:03.511570 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:03.511636 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:03.511989 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.512769 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.514191 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.514652 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.514756 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.515105 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.515192 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.515267 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:03.515379 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 05:23:03.515449 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260928 05:23:03.515827 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:03.510703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33913 (local address 127.8.124.194:57667)
0928 05:23:03.511283 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:03.511303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:03.511427 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:03.511640 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:03.511642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:03.511650 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:03.511877 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:03.511885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.512882 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.512888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:03.514077 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:03.514090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:03.514808 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:03.514814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:03.514937 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:03.514971 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:03.515280 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:03.515290 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:23:03.515355 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 05:23:03.515479 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":339,"threads_started":1}
W20260928 05:23:03.516800 11002 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:23:05.097484 10431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:05.097738 10431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:05.101644 10431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5648cd2cb2f4ee4b4b335e8477de7ce: Sending DeleteTablet for 3 replicas of tablet a13b7d7784484f9e9c87ba129461aedb
I20260928 05:23:05.102376 10706 tablet_service.cc:1514] Processing DeleteTablet for tablet a13b7d7784484f9e9c87ba129461aedb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44834
I20260928 05:23:05.102376 10568 tablet_service.cc:1514] Processing DeleteTablet for tablet a13b7d7784484f9e9c87ba129461aedb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:43166
I20260928 05:23:05.102502 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet a13b7d7784484f9e9c87ba129461aedb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:34246
I20260928 05:23:05.103157 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10502
I20260928 05:23:05.115118 11080 tablet_replica.cc:330] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: stopping tablet replica
I20260928 05:23:05.115478 11080 raft_consensus.cc:2229] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:05.115835 11080 raft_consensus.cc:2258] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:05.117470 11080 ts_tablet_manager.cc:1905] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:05.117718 11081 tablet_replica.cc:330] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: stopping tablet replica
I20260928 05:23:05.118021 11081 raft_consensus.cc:2229] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:05.119624 11082 tablet_replica.cc:330] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: stopping tablet replica
I20260928 05:23:05.119869 11082 raft_consensus.cc:2229] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:05.122081 11082 raft_consensus.cc:2258] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:05.118641 11081 raft_consensus.cc:2258] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:05.123176 11082 ts_tablet_manager.cc:1905] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:05.124115 11081 ts_tablet_manager.cc:1905] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:05.125123 11080 ts_tablet_manager.cc:1918] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:05.125295 11080 log.cc:1196] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/a13b7d7784484f9e9c87ba129461aedb
I20260928 05:23:05.125788 11080 ts_tablet_manager.cc:1939] T a13b7d7784484f9e9c87ba129461aedb P 09049f079f2e4189918a77b3ddf2bc0e: Deleting consensus metadata
I20260928 05:23:05.126892 10418 catalog_manager.cc:4841] TS 09049f079f2e4189918a77b3ddf2bc0e (127.8.124.194:43967): tablet a13b7d7784484f9e9c87ba129461aedb (table test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]) successfully deleted
I20260928 05:23:05.130695 11082 ts_tablet_manager.cc:1918] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:05.130915 11082 log.cc:1196] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/a13b7d7784484f9e9c87ba129461aedb
I20260928 05:23:05.131420 11082 ts_tablet_manager.cc:1939] T a13b7d7784484f9e9c87ba129461aedb P ac1a597683b745ecab8c4d13bbe88f09: Deleting consensus metadata
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:05.132525 10417 catalog_manager.cc:4841] TS ac1a597683b745ecab8c4d13bbe88f09 (127.8.124.195:38463): tablet a13b7d7784484f9e9c87ba129461aedb (table test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]) successfully deleted
I20260928 05:23:05.133567 11081 ts_tablet_manager.cc:1918] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:23:05.133802 11081 log.cc:1196] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/a13b7d7784484f9e9c87ba129461aedb
I20260928 05:23:05.134222 11081 ts_tablet_manager.cc:1939] T a13b7d7784484f9e9c87ba129461aedb P c626403d460c4018b8913140cfe0b351: Deleting consensus metadata
I20260928 05:23:05.135273 10416 catalog_manager.cc:4841] TS c626403d460c4018b8913140cfe0b351 (127.8.124.193:33913): tablet a13b7d7784484f9e9c87ba129461aedb (table test-table [id=3bdfab8d4ba549b3bcbf9a2e32c978a5]) successfully deleted
I20260928 05:23:05.139923 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:05.104343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.8.124.193:33913)
0928 05:23:05.121455 (+ 17112us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:05.121460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:05.121482 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:23:05.121541 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:05.121546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:05.121653 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:05.121755 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:05.123493 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.124534 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.126310 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.126901 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.126991 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:05.132941 (+ 5950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:05.132960 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:05.132984 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:05.133024 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:05.136969 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.137766 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.137776 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.137780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.137944 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.138281 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.138286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.138289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.138743 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:05.139010 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:05.139321 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 05:23:05.139536 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16981,"thread_start_us":17084,"threads_started":1}
W20260928 05:23:05.140590 10633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53152
W20260928 05:23:05.141194 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:05.141412 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10502
I20260928 05:23:05.159883 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10642
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:05.173882 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:05.160587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.8.124.194:43967)
0928 05:23:05.161039 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:05.161043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:05.161058 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:05.161109 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:05.161113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:05.161167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:05.161286 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:05.162478 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.163211 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.164377 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.164744 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.164832 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:05.168681 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:05.168714 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:05.168731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:05.168764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:05.171363 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.171926 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.171932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.171934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.172054 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.172441 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.172447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.172449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.172864 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:05.173097 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:05.173336 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 05:23:05.173504 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":213,"threads_started":1}
W20260928 05:23:05.174526 10772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33194
W20260928 05:23:05.175120 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10642: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:05.175261 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10642
I20260928 05:23:05.200237 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10781
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[10386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:05.214093 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:05.200936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.8.124.195:38463)
0928 05:23:05.201378 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:05.201383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:05.201398 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:05.201452 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:05.201455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:05.201516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:05.201644 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:05.202866 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.203576 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.204695 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:05.205056 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:05.205139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:05.209144 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:05.209159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:05.209175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:05.209211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:05.211671 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.212217 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.212224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.212226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.212337 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:05.212653 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:05.212656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:05.212658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:05.213010 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:05.213214 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:05.213557 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 05:23:05.213753 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
W20260928 05:23:05.214751 10911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33526
W20260928 05:23:05.215313 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10781: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:05.215478 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10781
I20260928 05:23:05.232378 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 10399
I20260928 05:23:05.233503 10471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:05.345187 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 10399
2026-09-28T05:23:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7802 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[11090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[11090](info): set up 2 sockets
Sep 28 05:23:05 dist-test-slave-515x krb5kdc[11090](info): commencing operation
krb5kdc: starting...
W20260928 05:23:06.329473 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:06 dist-test-slave-515x krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:06.499012 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37937
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:06.799670 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:06.799975 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:06.800105 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:06.807106 11103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:06.807212 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:06.807279 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:06.807344 11103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:06.807405 11103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:06.816983 11103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:37937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:06.818393 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:06.819705 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:06.831167 11109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:06.831255 11108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:06.831287 11111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:06.831880 11103 server_base.cc:1034] running on GCE node
I20260928 05:23:06.832612 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:06.834039 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:06.835273 11103 hybrid_clock.cc:648] HybridClock initialized: now 1790572986835183 us; error 36 us; skew 500 ppm
I20260928 05:23:06.835579 11103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:06 dist-test-slave-515x krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572986, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:06.838495 11103 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:06.839905 11103 webserver.cc:460] Webserver started at http://127.8.124.254:42519/ using document root <none> and password file <none>
I20260928 05:23:06.840549 11103 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:06.840656 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:06.840922 11103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:06.843744 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "757023b872014a8086dab2d66262ca88"
format_stamp: "Formatted at 2026-09-28 05:23:06 on dist-test-slave-515x"
server_key: "fa8bde384a579d7ae2d6aed9c8d2a22f"
server_key_iv: "f34f5e36c5937504e7d3bf77a2d17d80"
server_key_version: "encryptionkey@0"
I20260928 05:23:06.844475 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "757023b872014a8086dab2d66262ca88"
format_stamp: "Formatted at 2026-09-28 05:23:06 on dist-test-slave-515x"
server_key: "fa8bde384a579d7ae2d6aed9c8d2a22f"
server_key_iv: "f34f5e36c5937504e7d3bf77a2d17d80"
server_key_version: "encryptionkey@0"
I20260928 05:23:06.849274 11103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 05:23:06.852761 11118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:06.854276 11103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:06.854434 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "757023b872014a8086dab2d66262ca88"
format_stamp: "Formatted at 2026-09-28 05:23:06 on dist-test-slave-515x"
server_key: "fa8bde384a579d7ae2d6aed9c8d2a22f"
server_key_iv: "f34f5e36c5937504e7d3bf77a2d17d80"
server_key_version: "encryptionkey@0"
I20260928 05:23:06.854616 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:06.865337 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:06.869220 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:06.869473 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:06.885735 11103 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:37937
I20260928 05:23:06.885747 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:37937 every 8 connection(s)
I20260928 05:23:06.887041 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:06.890517 11170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:06.894109 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11103
I20260928 05:23:06.894301 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:06.894868 8691 external_mini_cluster.cc:1428] Setting key d0a1f412607db750c8fc84f3e2f88805
I20260928 05:23:06.900856 11170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88: Bootstrap starting.
I20260928 05:23:06.904265 11170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:06.905342 11170 log.cc:824] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:06.908356 11170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88: No bootstrap required, opened a new log
Sep 28 05:23:06 dist-test-slave-515x krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:06.916815 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:06.896670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.8.124.254:37937)
0928 05:23:06.897417 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:06.897438 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:06.897485 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:23:06.899405 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:06.899435 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:06.899825 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:06.900134 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:06.901862 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:06.903146 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:06.904829 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:06.905490 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:06.905659 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:06.910133 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:06.910143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:06.910160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:06.910213 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:06.913129 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:06.913878 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:06.913883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:06.913889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:06.914043 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:06.914400 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:06.914403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:06.914404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:06.914888 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:06.915090 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:06.915527 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 05:23:06.915858 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260928 05:23:06.916755 11170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } }
I20260928 05:23:06.917156 11170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:06.917264 11170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757023b872014a8086dab2d66262ca88, State: Initialized, Role: FOLLOWER
I20260928 05:23:06.917898 11170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } }
I20260928 05:23:06.918164 11170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:06.918290 11170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:06.918455 11170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:06.920644 11170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } }
I20260928 05:23:06.921072 11170 leader_election.cc:302] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 757023b872014a8086dab2d66262ca88; no voters:
I20260928 05:23:06.921542 11170 leader_election.cc:288] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:06.921665 11175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:06.922266 11175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [term 1 LEADER]: Becoming Leader. State: Replica: 757023b872014a8086dab2d66262ca88, State: Running, Role: LEADER
I20260928 05:23:06.922765 11175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } }
I20260928 05:23:06.923228 11170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:06.925153 11177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 757023b872014a8086dab2d66262ca88. Latest consensus state: current_term: 1 leader_uuid: "757023b872014a8086dab2d66262ca88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } } }
I20260928 05:23:06.925482 11177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:06.926873 11176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "757023b872014a8086dab2d66262ca88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757023b872014a8086dab2d66262ca88" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37937 } } }
I20260928 05:23:06.927261 11176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:06.928221 11183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:06.932862 11183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:06.941974 11183 catalog_manager.cc:1347] Generated new cluster ID: c3bfcc08ccab472bbcd9f63c98c9a24d
I20260928 05:23:06.942095 11183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:06.949317 11183 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:06.951164 11183 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:06.963771 11183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 757023b872014a8086dab2d66262ca88: Generated new TSK 0
I20260928 05:23:06.964751 11183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:07.017071 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:07.336697 11196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:07.337026 11196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:07.337172 11196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:07.344682 11196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:07.344820 11196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:07.344980 11196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:07.355026 11196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:07.356302 11196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:07.357649 11196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:07.369900 11202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:07.369880 11204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:07.370127 11201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:07.370667 11196 server_base.cc:1034] running on GCE node
I20260928 05:23:07.371184 11196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:07.371999 11196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:07.373186 11196 hybrid_clock.cc:648] HybridClock initialized: now 1790572987373102 us; error 75 us; skew 500 ppm
I20260928 05:23:07.373445 11196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:07 dist-test-slave-515x krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:07.376430 11196 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:07.377904 11196 webserver.cc:460] Webserver started at http://127.8.124.193:33427/ using document root <none> and password file <none>
I20260928 05:23:07.378588 11196 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:07.378702 11196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:07.379014 11196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:07.382047 11196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "d242ab27851347469ebfa821968f5447"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "97b9261a2d9116561ec063eb47c9f0b8"
server_key_iv: "884015bf8402fcca06d6b88466862d8b"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.382834 11196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "d242ab27851347469ebfa821968f5447"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "97b9261a2d9116561ec063eb47c9f0b8"
server_key_iv: "884015bf8402fcca06d6b88466862d8b"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.387780 11196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:23:07.391120 11211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.392385 11196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:23:07.392525 11196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "d242ab27851347469ebfa821968f5447"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "97b9261a2d9116561ec063eb47c9f0b8"
server_key_iv: "884015bf8402fcca06d6b88466862d8b"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.392688 11196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:07.421012 11196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:07.425079 11196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:07.425330 11196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:07.426393 11196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:07.427873 11196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:07.427948 11196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.428063 11196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:07.428114 11196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.456954 11196 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:36347
I20260928 05:23:07.457005 11323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:36347 every 8 connection(s)
I20260928 05:23:07.458390 11196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:07.466418 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11196
I20260928 05:23:07.466624 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:07.467469 8691 external_mini_cluster.cc:1428] Setting key bd930c3007bb3c7c34ea49c16de3da92
Sep 28 05:23:07 dist-test-slave-515x krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:07.478571 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:07.461017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:38309 (local address 127.8.124.254:37937)
0928 05:23:07.461461 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:07.461466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:07.462612 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 05:23:07.464481 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:07.464485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:07.464548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:07.464643 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:07.466178 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.467332 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.469336 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.469905 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.470031 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:07.473201 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:07.473207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:07.473211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:07.473247 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:07.475739 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:07.476544 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:07.476547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:07.476550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:07.476659 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:07.477047 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:07.477050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:07.477052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:07.477283 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:07.477390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:07.478145 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 05:23:07.478297 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":243,"threads_started":1}
I20260928 05:23:07.479873 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:07.461699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37937 (local address 127.8.124.193:38309)
0928 05:23:07.462454 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:07.462497 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:07.464234 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:07.464829 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:07.464837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:07.465387 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:07.465990 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:07.466006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.467526 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.467530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:07.469176 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:07.469190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.470096 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.470101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:07.470328 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:07.470982 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:07.471003 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:07.472996 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:07.475877 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:07.475886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:07.475904 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:07.476448 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:07.476743 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:07.476747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:07.476750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:07.476959 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:07.477394 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:07.477401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:07.477806 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 05:23:07.478074 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 05:23:07.478568 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":278,"threads_started":1}
I20260928 05:23:07.481359 11324 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37937
I20260928 05:23:07.481700 11324 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:07.482533 11324 heartbeater.cc:511] Master 127.8.124.254:37937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:23:07.484589 11135 ts_manager.cc:194] Registered new tserver with Master: d242ab27851347469ebfa821968f5447 (127.8.124.193:36347)
I20260928 05:23:07.486136 11135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:38309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:07.506392 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:07.808121 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:07.808387 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:07.808506 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:07.815500 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:07.815599 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:07.815750 11330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:07.825428 11330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:07.826673 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:07.828078 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:07.840337 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:07.840435 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:07.840436 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:07.841149 11330 server_base.cc:1034] running on GCE node
I20260928 05:23:07.841671 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:07.842486 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:07.843677 11330 hybrid_clock.cc:648] HybridClock initialized: now 1790572987843627 us; error 42 us; skew 500 ppm
I20260928 05:23:07.843936 11330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:07 dist-test-slave-515x krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:07.846788 11330 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:07.848067 11330 webserver.cc:460] Webserver started at http://127.8.124.194:45797/ using document root <none> and password file <none>
I20260928 05:23:07.848697 11330 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:07.848786 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:07.849017 11330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:07.851790 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "4ea53077923d4739a65fe43e9f3a1248"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "4c8fe5fae1db54e88925368ba9f76fc8"
server_key_iv: "e4423b6fdba9de84040b39cc9cc72f3c"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.852483 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "4ea53077923d4739a65fe43e9f3a1248"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "4c8fe5fae1db54e88925368ba9f76fc8"
server_key_iv: "e4423b6fdba9de84040b39cc9cc72f3c"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.857087 11330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 05:23:07.860554 11345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.861882 11330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:07.862025 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "4ea53077923d4739a65fe43e9f3a1248"
format_stamp: "Formatted at 2026-09-28 05:23:07 on dist-test-slave-515x"
server_key: "4c8fe5fae1db54e88925368ba9f76fc8"
server_key_iv: "e4423b6fdba9de84040b39cc9cc72f3c"
server_key_version: "encryptionkey@0"
I20260928 05:23:07.862193 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:07.877393 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:07.881240 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:07.881472 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:07.882321 11330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:07.883479 11330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:07.883551 11330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.883668 11330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:07.883718 11330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:07.915337 11330 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:41851
I20260928 05:23:07.915373 11457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:41851 every 8 connection(s)
I20260928 05:23:07.916518 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:07.921981 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11330
I20260928 05:23:07.922165 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:07.922695 8691 external_mini_cluster.cc:1428] Setting key 66a5cfd0cbf17ec2a30f1ca183dd45e2
Sep 28 05:23:07 dist-test-slave-515x krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:07.936022 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:07.918772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:44513 (local address 127.8.124.254:37937)
0928 05:23:07.918936 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:07.918940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:07.920029 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 05:23:07.921682 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:07.921686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:07.921774 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:07.921858 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:07.923775 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.924799 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.926961 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.927394 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.927484 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:07.931024 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:07.931029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:07.931032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:07.931062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:07.933331 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:07.934011 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:07.934016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:07.934020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:07.934129 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:07.934502 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:07.934506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:07.934508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:07.934738 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:07.934895 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:07.935492 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 05:23:07.935695 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:23:07.937009 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:07.919211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37937 (local address 127.8.124.194:44513)
0928 05:23:07.919876 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:07.919913 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:07.921416 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:07.922129 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:07.922140 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:07.922831 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:07.923585 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:07.923607 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.924961 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.924966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:07.926835 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:07.926850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:07.927597 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:07.927603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:07.927870 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:07.928543 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:07.928573 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:07.930831 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:07.933447 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:07.933455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:07.933472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:07.933909 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:07.934222 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:07.934226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:07.934230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:07.934398 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:07.934850 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:07.934858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:07.935177 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 05:23:07.935395 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:23:07.935885 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":245,"threads_started":1}
I20260928 05:23:07.938361 11458 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37937
I20260928 05:23:07.938621 11458 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:23:07.939301 11458 heartbeater.cc:511] Master 127.8.124.254:37937 requested a full tablet report, sending...
I20260928 05:23:07.940651 11135 ts_manager.cc:194] Registered new tserver with Master: 4ea53077923d4739a65fe43e9f3a1248 (127.8.124.194:41851)
I20260928 05:23:07.941460 11135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:44513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:07.962977 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:08.266707 11463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:08.266980 11463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:08.267107 11463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:08.274114 11463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:08.274214 11463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:08.274355 11463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:08.283799 11463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:08.284981 11463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:08.286211 11463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:08.298501 11469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:08.298578 11468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:08.298610 11471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:08.299115 11463 server_base.cc:1034] running on GCE node
I20260928 05:23:08.299700 11463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:08.300588 11463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:08.301800 11463 hybrid_clock.cc:648] HybridClock initialized: now 1790572988301742 us; error 41 us; skew 500 ppm
I20260928 05:23:08.302117 11463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:08 dist-test-slave-515x krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:08.308331 11463 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:08.309767 11463 webserver.cc:460] Webserver started at http://127.8.124.195:39375/ using document root <none> and password file <none>
I20260928 05:23:08.310431 11463 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:08.310521 11463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:08.310776 11463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:08.313472 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "f92052078f584286afd28a2c529e8494"
format_stamp: "Formatted at 2026-09-28 05:23:08 on dist-test-slave-515x"
server_key: "d73a2cd0bb6dce079b6b1eb591fe0928"
server_key_iv: "1898d9d587b924d7b0e31e15bf2afae8"
server_key_version: "encryptionkey@0"
I20260928 05:23:08.314222 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "f92052078f584286afd28a2c529e8494"
format_stamp: "Formatted at 2026-09-28 05:23:08 on dist-test-slave-515x"
server_key: "d73a2cd0bb6dce079b6b1eb591fe0928"
server_key_iv: "1898d9d587b924d7b0e31e15bf2afae8"
server_key_version: "encryptionkey@0"
I20260928 05:23:08.318876 11463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:23:08.321969 11478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:08.323290 11463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:23:08.323421 11463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "f92052078f584286afd28a2c529e8494"
format_stamp: "Formatted at 2026-09-28 05:23:08 on dist-test-slave-515x"
server_key: "d73a2cd0bb6dce079b6b1eb591fe0928"
server_key_iv: "1898d9d587b924d7b0e31e15bf2afae8"
server_key_version: "encryptionkey@0"
I20260928 05:23:08.323578 11463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:08.337067 11463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:08.340998 11463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:08.341233 11463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:08.342245 11463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:08.343626 11463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:08.343698 11463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:08.343796 11463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:08.343844 11463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:08.372854 11463 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:46537
I20260928 05:23:08.372886 11590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:46537 every 8 connection(s)
I20260928 05:23:08.374095 11463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:08.378273 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11463
I20260928 05:23:08.378470 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:08.379010 8691 external_mini_cluster.cc:1428] Setting key fd1006fa9147e42db141349fbbd42302
Sep 28 05:23:08 dist-test-slave-515x krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:08.393399 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:08.376347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47731 (local address 127.8.124.254:37937)
0928 05:23:08.376562 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:08.376568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:08.377841 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 05:23:08.380219 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:08.380224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:08.380308 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:08.380399 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:08.382265 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:08.382987 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:08.384516 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:08.384909 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:08.385016 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:08.388178 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:08.388182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:08.388185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:08.388215 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:08.390697 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:08.391336 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:08.391339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:08.391342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:08.391470 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:08.391830 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:08.391833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:08.391835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:08.392117 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:08.392224 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:08.392918 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 05:23:08.393122 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:23:08.394500 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:08.376933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37937 (local address 127.8.124.195:47731)
0928 05:23:08.377670 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:08.377721 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:08.379943 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:08.380974 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:08.380983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:08.381485 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:08.382112 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:08.382129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:08.383129 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:08.383133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:08.384386 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:08.384397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:08.385115 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:08.385122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:08.385344 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:08.385961 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:08.385982 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:08.387991 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:08.390830 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:08.390836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:08.390848 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:08.391223 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:08.391551 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:08.391553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:08.391556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:08.391726 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:08.392235 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:08.392242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:08.392585 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 05:23:08.392795 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 05:23:08.393275 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
I20260928 05:23:08.395645 11591 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37937
I20260928 05:23:08.395962 11591 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:08.396618 11591 heartbeater.cc:511] Master 127.8.124.254:37937 requested a full tablet report, sending...
I20260928 05:23:08.397907 11135 ts_manager.cc:194] Registered new tserver with Master: f92052078f584286afd28a2c529e8494 (127.8.124.195:46537)
I20260928 05:23:08.398691 11135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:47731
I20260928 05:23:08.406207 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:23:08.432783 11326 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:08.430417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.8.124.254:37937)
0928 05:23:08.430580 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:08.430585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:08.431037 (+ 452us) server_negotiation.cc:407] Connection header received
0928 05:23:08.431207 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:08.431211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:08.431281 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:08.431363 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:08.432545 (+ 1182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38860: BlockingRecv error: recv got EOF from 127.0.0.1:38860 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:23:08.435477 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11196
W20260928 05:23:08.438091 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11196: Not authorized: Client connection negotiation failed: client connection to 127.8.124.193:36347: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:23:08.438271 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11196
I20260928 05:23:08.450251 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11330
W20260928 05:23:08.452484 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11330: Not authorized: Client connection negotiation failed: client connection to 127.8.124.194:41851: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:23:08.452605 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11330
I20260928 05:23:08.464552 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11463
W20260928 05:23:08.467306 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11463: Not authorized: Client connection negotiation failed: client connection to 127.8.124.195:46537: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:23:08.467413 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11463
I20260928 05:23:08.478978 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11103
I20260928 05:23:08.480026 11165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:08.578537 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11103
2026-09-28T05:23:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3230 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:08 dist-test-slave-515x krb5kdc[11608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:08 dist-test-slave-515x krb5kdc[11608](info): set up 2 sockets
Sep 28 05:23:08 dist-test-slave-515x krb5kdc[11608](info): commencing operation
krb5kdc: starting...
W20260928 05:23:09.561566 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:09 dist-test-slave-515x krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:09.732501 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43115
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:43115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:10.032433 11621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:10.032698 11621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:10.032799 11621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:10.039709 11621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:10.039804 11621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:10.039856 11621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:10.039909 11621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:10.039954 11621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:10.049186 11621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:43115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:10.050395 11621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:10.051578 11621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:10.062107 11626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:10.062134 11627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:10.062079 11629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:10.063192 11621 server_base.cc:1034] running on GCE node
I20260928 05:23:10.063925 11621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:10.065265 11621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:10.066505 11621 hybrid_clock.cc:648] HybridClock initialized: now 1790572990066440 us; error 58 us; skew 500 ppm
I20260928 05:23:10.066826 11621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:10 dist-test-slave-515x krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572990, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:10.069698 11621 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:10.071028 11621 webserver.cc:460] Webserver started at http://127.8.124.254:46133/ using document root <none> and password file <none>
I20260928 05:23:10.071653 11621 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:10.071765 11621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:10.072036 11621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:10.074855 11621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "b57f4f3df6024a578adfb2d5e56b81b5"
format_stamp: "Formatted at 2026-09-28 05:23:10 on dist-test-slave-515x"
server_key: "72d88ec394f6bd2d67daf3f9e8711aea"
server_key_iv: "a9b0b6c7ca0c3675fbbf4971038cf110"
server_key_version: "encryptionkey@0"
I20260928 05:23:10.075574 11621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "b57f4f3df6024a578adfb2d5e56b81b5"
format_stamp: "Formatted at 2026-09-28 05:23:10 on dist-test-slave-515x"
server_key: "72d88ec394f6bd2d67daf3f9e8711aea"
server_key_iv: "a9b0b6c7ca0c3675fbbf4971038cf110"
server_key_version: "encryptionkey@0"
I20260928 05:23:10.080286 11621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:10.083424 11636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:10.084619 11621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:10.084765 11621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "b57f4f3df6024a578adfb2d5e56b81b5"
format_stamp: "Formatted at 2026-09-28 05:23:10 on dist-test-slave-515x"
server_key: "72d88ec394f6bd2d67daf3f9e8711aea"
server_key_iv: "a9b0b6c7ca0c3675fbbf4971038cf110"
server_key_version: "encryptionkey@0"
I20260928 05:23:10.084929 11621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:10.102265 11621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:10.105827 11621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:10.106062 11621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:10.121955 11621 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:43115
I20260928 05:23:10.121984 11687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:43115 every 8 connection(s)
I20260928 05:23:10.123224 11621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:10.126267 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11621
I20260928 05:23:10.126456 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:10.127035 11688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:10.127275 8691 external_mini_cluster.cc:1428] Setting key 58f2a4e9bedc97074df0d9d3c25b30c0
I20260928 05:23:10.140575 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5: Bootstrap starting.
Sep 28 05:23:10 dist-test-slave-515x krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:10.144034 11688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:10.145251 11688 log.cc:824] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:10.148398 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5: No bootstrap required, opened a new log
I20260928 05:23:10.150183 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:10.129082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.8.124.254:43115)
0928 05:23:10.129865 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:10.129880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:10.129931 (+ 51us) server_negotiation.cc:407] Connection header received
0928 05:23:10.131751 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:10.131779 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:10.132287 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:10.132719 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:10.134090 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:10.135968 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:10.137262 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:10.138349 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:10.138523 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:10.142114 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:10.142130 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:10.142148 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:10.142189 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:10.145707 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:10.146530 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:10.146537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:10.146546 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:10.146730 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:10.147061 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:10.147065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:10.147068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:10.147725 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:10.147994 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:10.148351 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 05:23:10.148756 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":304,"threads_started":1}
I20260928 05:23:10.154105 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:23:10.156701 11688 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } }
I20260928 05:23:10.157007 11688 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:10.157101 11688 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57f4f3df6024a578adfb2d5e56b81b5, State: Initialized, Role: FOLLOWER
I20260928 05:23:10.157630 11688 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } }
I20260928 05:23:10.157832 11688 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:10.157953 11688 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:10.158089 11688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:10.160048 11688 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } }
I20260928 05:23:10.160496 11688 leader_election.cc:302] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57f4f3df6024a578adfb2d5e56b81b5; no voters:
I20260928 05:23:10.161062 11688 leader_election.cc:288] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:10.161158 11694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:10.162212 11694 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [term 1 LEADER]: Becoming Leader. State: Replica: b57f4f3df6024a578adfb2d5e56b81b5, State: Running, Role: LEADER
I20260928 05:23:10.162775 11694 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } }
I20260928 05:23:10.163198 11688 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:10.165508 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } } }
I20260928 05:23:10.165912 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:10.165385 11696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57f4f3df6024a578adfb2d5e56b81b5. Latest consensus state: current_term: 1 leader_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f4f3df6024a578adfb2d5e56b81b5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43115 } } }
I20260928 05:23:10.166209 11696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:10.177037 11703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:10.182225 11703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:10.191885 11703 catalog_manager.cc:1347] Generated new cluster ID: 9f097920b7074fc2b1da4bc66782db88
I20260928 05:23:10.192003 11703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:10.196471 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:10.188666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.8.124.254:43115)
0928 05:23:10.188837 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:10.188843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:10.189397 (+ 554us) server_negotiation.cc:407] Connection header received
0928 05:23:10.189590 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:10.189615 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:10.189717 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:10.189823 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:10.191088 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:10.192168 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:10.193446 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:10.193960 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:10.194072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:10.195129 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:10.195134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:10.195137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:10.195737 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:10.195779 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:10.195809 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:10.195899 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:10.196039 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 05:23:10.196136 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:23:10.227468 11703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:10.228770 11703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:10.240651 11703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57f4f3df6024a578adfb2d5e56b81b5: Generated new TSK 0
I20260928 05:23:10.241499 11703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:23:10.250939 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11621
I20260928 05:23:10.252344 11683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:10.340683 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11621
2026-09-28T05:23:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1757 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:10 dist-test-slave-515x krb5kdc[11722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:10 dist-test-slave-515x krb5kdc[11722](info): set up 2 sockets
Sep 28 05:23:10 dist-test-slave-515x krb5kdc[11722](info): commencing operation
krb5kdc: starting...
W20260928 05:23:11.321550 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:11 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:11.497123 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:34117
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:11.801493 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:11.801816 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:11.801952 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:11.808966 11735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:11.809062 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:11.809116 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:11.809190 11735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:11.809238 11735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:11.818831 11735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:11.820024 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:11.821210 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:11.831804 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:11.831873 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:11.831873 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:11.832765 11735 server_base.cc:1034] running on GCE node
I20260928 05:23:11.833474 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:11.834748 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:11.835973 11735 hybrid_clock.cc:648] HybridClock initialized: now 1790572991835930 us; error 26 us; skew 500 ppm
I20260928 05:23:11.836331 11735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:11 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572991, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:11.839124 11735 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:11.840467 11735 webserver.cc:460] Webserver started at http://127.8.124.254:37643/ using document root <none> and password file <none>
I20260928 05:23:11.841063 11735 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:11.841173 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:11.841444 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:11.844184 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "490929d055f9454187e3d6508e4aed41"
format_stamp: "Formatted at 2026-09-28 05:23:11 on dist-test-slave-515x"
server_key: "3600ef14faf8c162e56c0ae3b583601a"
server_key_iv: "4bd569ab2168ca97e8158180a58f32f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:11.844875 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "490929d055f9454187e3d6508e4aed41"
format_stamp: "Formatted at 2026-09-28 05:23:11 on dist-test-slave-515x"
server_key: "3600ef14faf8c162e56c0ae3b583601a"
server_key_iv: "4bd569ab2168ca97e8158180a58f32f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:11.849633 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 05:23:11.852836 11750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:11.854094 11735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:23:11.854246 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "490929d055f9454187e3d6508e4aed41"
format_stamp: "Formatted at 2026-09-28 05:23:11 on dist-test-slave-515x"
server_key: "3600ef14faf8c162e56c0ae3b583601a"
server_key_iv: "4bd569ab2168ca97e8158180a58f32f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:11.854414 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:11.870682 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:11.874676 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:11.874948 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:11.891358 11735 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:34117
I20260928 05:23:11.891376 11801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:34117 every 8 connection(s)
I20260928 05:23:11.892632 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:11.896140 11802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:11.902976 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11735
I20260928 05:23:11.903167 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:11.903756 8691 external_mini_cluster.cc:1428] Setting key 1c2ac53ed0d2eb48cf4620c99fa94a30
I20260928 05:23:11.905714 11802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41: Bootstrap starting.
I20260928 05:23:11.908810 11802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:11.909808 11802 log.cc:824] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:11.912367 11802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41: No bootstrap required, opened a new log
Sep 28 05:23:11 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:11.919826 11802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } }
I20260928 05:23:11.920168 11802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:11.920264 11802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490929d055f9454187e3d6508e4aed41, State: Initialized, Role: FOLLOWER
I20260928 05:23:11.920812 11802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } }
I20260928 05:23:11.921036 11802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:11.921169 11802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:11.921332 11802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:11.923918 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:11.905713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.8.124.254:34117)
0928 05:23:11.906442 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:11.906453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:11.906492 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:23:11.908102 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:11.908124 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:11.908480 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:11.908830 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:11.910202 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:11.911485 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:11.912725 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:11.913357 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:11.913509 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:11.917335 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:11.917343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:11.917357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:11.917396 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:11.920179 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:11.920818 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:11.920823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:11.920829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:11.920972 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:11.921381 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:11.921385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:11.921387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:11.921986 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:11.922213 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:11.922621 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 05:23:11.922953 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
I20260928 05:23:11.923877 11802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } }
I20260928 05:23:11.924340 11802 leader_election.cc:302] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 490929d055f9454187e3d6508e4aed41; no voters:
I20260928 05:23:11.924916 11802 leader_election.cc:288] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:11.925017 11807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:11.926044 11807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [term 1 LEADER]: Becoming Leader. State: Replica: 490929d055f9454187e3d6508e4aed41, State: Running, Role: LEADER
I20260928 05:23:11.926643 11807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } }
I20260928 05:23:11.927065 11802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:11.930742 11808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "490929d055f9454187e3d6508e4aed41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } } }
I20260928 05:23:11.930814 11809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490929d055f9454187e3d6508e4aed41. Latest consensus state: current_term: 1 leader_uuid: "490929d055f9454187e3d6508e4aed41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490929d055f9454187e3d6508e4aed41" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 34117 } } }
I20260928 05:23:11.931079 11808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:11.931121 11809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:11.932422 11817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:11.937565 11817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:11.945629 11817 catalog_manager.cc:1347] Generated new cluster ID: 210e8f83a6ce48fdb2fd9035a136e59f
I20260928 05:23:11.945725 11817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:11.968487 11817 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:11.969861 11817 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:11.979285 11817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 490929d055f9454187e3d6508e4aed41: Generated new TSK 0
I20260928 05:23:11.980087 11817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:12.022670 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:12.352491 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:12.352769 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:12.352914 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:12.360258 11828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:12.360386 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:12.360531 11828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:12.370162 11828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:12.371408 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:12.372716 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:12.384611 11833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:12.384591 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:12.384661 11834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:12.385495 11828 server_base.cc:1034] running on GCE node
I20260928 05:23:12.386042 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:12.386914 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:12.388094 11828 hybrid_clock.cc:648] HybridClock initialized: now 1790572992388056 us; error 31 us; skew 500 ppm
I20260928 05:23:12.388368 11828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:12 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572992, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:12.391284 11828 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:12.392575 11828 webserver.cc:460] Webserver started at http://127.8.124.193:45467/ using document root <none> and password file <none>
I20260928 05:23:12.393159 11828 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:12.393252 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:12.393494 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:12.396214 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "b1ff9b0de97c441b95346080c01caaa1"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "a2e26977fb2d53a9173629c795ada26f"
server_key_iv: "046e9f6e5342f9c1871104435c4e120f"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.396903 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "b1ff9b0de97c441b95346080c01caaa1"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "a2e26977fb2d53a9173629c795ada26f"
server_key_iv: "046e9f6e5342f9c1871104435c4e120f"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.401641 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260928 05:23:12.404812 11843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.406073 11828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:23:12.406198 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "b1ff9b0de97c441b95346080c01caaa1"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "a2e26977fb2d53a9173629c795ada26f"
server_key_iv: "046e9f6e5342f9c1871104435c4e120f"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.406366 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:12.428753 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:12.432808 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:12.433063 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:12.433975 11828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:12.435163 11828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:12.435237 11828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.435352 11828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:12.435400 11828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.463920 11828 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:35439
I20260928 05:23:12.463964 11955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:35439 every 8 connection(s)
I20260928 05:23:12.465106 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:12.470671 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11828
I20260928 05:23:12.470834 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:12.471447 8691 external_mini_cluster.cc:1428] Setting key 88c8435dd10779833d1c03edbf878845
Sep 28 05:23:12 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572992, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:12.483482 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:12.467358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:54475 (local address 127.8.124.254:34117)
0928 05:23:12.467742 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:12.467747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:12.468605 (+ 858us) server_negotiation.cc:407] Connection header received
0928 05:23:12.470427 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:12.470433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:12.470512 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:12.470602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:12.472120 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.473091 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.474593 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.474965 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.475065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:12.478366 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:12.478370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:12.478374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:12.478412 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:12.480706 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:12.481349 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:12.481354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:12.481357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:12.481488 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:12.481912 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:12.481915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:12.481917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:12.482167 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:12.482290 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:12.482942 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 05:23:12.483152 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":200,"threads_started":1}
I20260928 05:23:12.484377 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:12.467808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:34117 (local address 127.8.124.193:54475)
0928 05:23:12.468461 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:12.468503 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:12.470148 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:12.470866 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:12.470876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:12.471389 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:12.471965 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:12.471982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.473225 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.473229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:12.474467 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:12.474478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.475159 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.475166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:12.475380 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:12.475994 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:12.476014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:12.478164 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:12.480828 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:12.480836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:12.480853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:12.481232 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:12.481591 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:12.481595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:12.481630 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:12.481803 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:12.482283 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:12.482290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:12.482609 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 05:23:12.482829 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 05:23:12.483304 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":236,"threads_started":1}
I20260928 05:23:12.485791 11956 heartbeater.cc:347] Connected to a master server at 127.8.124.254:34117
I20260928 05:23:12.486100 11956 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:12.486923 11956 heartbeater.cc:511] Master 127.8.124.254:34117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:23:12.488940 11767 ts_manager.cc:194] Registered new tserver with Master: b1ff9b0de97c441b95346080c01caaa1 (127.8.124.193:35439)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:23:12.490645 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:54475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:12.508476 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:12.812855 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:12.813117 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:12.813206 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:12.820302 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:12.820394 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:12.820505 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:12.829952 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:12.831120 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:12.832291 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:12.844089 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:12.844120 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:12.844062 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:12.844712 11962 server_base.cc:1034] running on GCE node
I20260928 05:23:12.845257 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:12.846151 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:12.847383 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790572992847324 us; error 50 us; skew 500 ppm
I20260928 05:23:12.847684 11962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:12 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572992, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:12.850704 11962 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:12.852092 11962 webserver.cc:460] Webserver started at http://127.8.124.194:33891/ using document root <none> and password file <none>
I20260928 05:23:12.852746 11962 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:12.852864 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:12.853142 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:12.855981 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "39ddf64c264a458288a1e4f6518bd020"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "d616405c49ac195b87be3d5fae104abf"
server_key_iv: "4e25e34c47a500f3b787b1fbde28acb7"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.856694 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "39ddf64c264a458288a1e4f6518bd020"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "d616405c49ac195b87be3d5fae104abf"
server_key_iv: "4e25e34c47a500f3b787b1fbde28acb7"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.861584 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 05:23:12.864759 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.866047 11962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:23:12.866180 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "39ddf64c264a458288a1e4f6518bd020"
format_stamp: "Formatted at 2026-09-28 05:23:12 on dist-test-slave-515x"
server_key: "d616405c49ac195b87be3d5fae104abf"
server_key_iv: "4e25e34c47a500f3b787b1fbde28acb7"
server_key_version: "encryptionkey@0"
I20260928 05:23:12.866341 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:12.894651 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:12.898593 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:12.899037 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:12.899984 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:12.901324 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:12.901397 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.901506 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:12.901566 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:12.930172 11962 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45271
I20260928 05:23:12.930220 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45271 every 8 connection(s)
I20260928 05:23:12.931371 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:12.934257 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 11962
I20260928 05:23:12.934403 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:12.934834 8691 external_mini_cluster.cc:1428] Setting key fc3c6a7663863371ad941775843a6095
Sep 28 05:23:12 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572992, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:12.951056 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:12.933747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:59899 (local address 127.8.124.254:34117)
0928 05:23:12.933960 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:12.933966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:12.935205 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 05:23:12.937166 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:12.937172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:12.937264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:12.937380 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:12.938753 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.939684 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.941164 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.941733 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.941831 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:12.945704 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:12.945708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:12.945712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:12.945741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:12.948068 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:12.948875 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:12.948879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:12.948881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:12.948996 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:12.949431 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:12.949434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:12.949436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:12.949705 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:12.949813 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:12.950640 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 05:23:12.950796 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:23:12.952427 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:12.934287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:34117 (local address 127.8.124.194:59899)
0928 05:23:12.935045 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:12.935089 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:12.936921 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:12.937537 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:12.937546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:12.938047 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:12.938600 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:12.938617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.939819 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.939823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:12.941049 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:12.941060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:12.941940 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:12.941947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:12.942242 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:12.942920 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:12.942946 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:12.945459 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:12.948209 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:12.948227 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:12.948245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:12.948775 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:12.949089 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:12.949093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:12.949097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:12.949337 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:12.949821 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:12.949828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:12.950240 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 05:23:12.950545 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 05:23:12.951048 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":331,"threads_started":1}
I20260928 05:23:12.953742 12090 heartbeater.cc:347] Connected to a master server at 127.8.124.254:34117
I20260928 05:23:12.954020 12090 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:12.954854 12090 heartbeater.cc:511] Master 127.8.124.254:34117 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
I20260928 05:23:12.956382 11767 ts_manager.cc:194] Registered new tserver with Master: 39ddf64c264a458288a1e4f6518bd020 (127.8.124.194:45271)
I20260928 05:23:12.957151 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:59899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:12.974984 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:13.276271 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:13.276532 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:13.276650 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:13.283594 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:13.283694 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:13.283838 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:13.293072 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:13.294292 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:13.295491 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:13.307364 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:13.307397 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:13.307364 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:13.308139 12095 server_base.cc:1034] running on GCE node
I20260928 05:23:13.308652 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:13.309473 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:13.310650 12095 hybrid_clock.cc:648] HybridClock initialized: now 1790572993310601 us; error 44 us; skew 500 ppm
I20260928 05:23:13.310935 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:13.316669 12095 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:13.318024 12095 webserver.cc:460] Webserver started at http://127.8.124.195:37547/ using document root <none> and password file <none>
I20260928 05:23:13.318607 12095 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:13.318698 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:13.318938 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:13.321581 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "6a7a5f612ad645b9b29a4b0330c5b26e"
format_stamp: "Formatted at 2026-09-28 05:23:13 on dist-test-slave-515x"
server_key: "06c8f57cd12f5e51c9d85f3d3bb8391d"
server_key_iv: "7dc5d156524e050085b7f6a1eb9d4114"
server_key_version: "encryptionkey@0"
I20260928 05:23:13.322329 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "6a7a5f612ad645b9b29a4b0330c5b26e"
format_stamp: "Formatted at 2026-09-28 05:23:13 on dist-test-slave-515x"
server_key: "06c8f57cd12f5e51c9d85f3d3bb8391d"
server_key_iv: "7dc5d156524e050085b7f6a1eb9d4114"
server_key_version: "encryptionkey@0"
I20260928 05:23:13.327003 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:23:13.330180 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:13.331461 12095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:13.331583 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "6a7a5f612ad645b9b29a4b0330c5b26e"
format_stamp: "Formatted at 2026-09-28 05:23:13 on dist-test-slave-515x"
server_key: "06c8f57cd12f5e51c9d85f3d3bb8391d"
server_key_iv: "7dc5d156524e050085b7f6a1eb9d4114"
server_key_version: "encryptionkey@0"
I20260928 05:23:13.331729 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:13.347931 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:13.351728 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:13.351953 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:13.352756 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:13.353958 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:13.354024 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:13.354110 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:13.354151 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:13.382710 12095 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:33667
I20260928 05:23:13.382752 12222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:33667 every 8 connection(s)
I20260928 05:23:13.383883 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:13.391297 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12095
I20260928 05:23:13.391462 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:13.391875 8691 external_mini_cluster.cc:1428] Setting key 2ce2df56fb05747be3f2751711921337
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:13.401566 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.386195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:43229 (local address 127.8.124.254:34117)
0928 05:23:13.386343 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:13.386348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:13.387446 (+ 1098us) server_negotiation.cc:407] Connection header received
0928 05:23:13.389081 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:13.389085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:13.389149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:13.389232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:13.390646 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.391380 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.393092 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.393499 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.393642 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.396881 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:13.396886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:13.396889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:13.396920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:13.399256 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.399873 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.399876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.399879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.399977 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.400280 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.400296 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.400297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.400542 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:13.400640 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:13.401179 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 05:23:13.401333 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:13.402513 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.386645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:34117 (local address 127.8.124.195:43229)
0928 05:23:13.387294 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:13.387329 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:13.388856 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:13.389411 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:13.389419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:13.389951 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:13.390503 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:13.390520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.391549 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.391555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:13.392933 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:13.392946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.393759 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.393767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:13.394044 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.394705 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:13.394730 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:13.396700 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:13.399398 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:13.399404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:13.399415 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:13.399787 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:13.400047 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:13.400050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:13.400052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:13.400209 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:13.400627 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:13.400632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:13.400921 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 05:23:13.401132 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 05:23:13.401482 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":248,"threads_started":1}
I20260928 05:23:13.403802 12223 heartbeater.cc:347] Connected to a master server at 127.8.124.254:34117
I20260928 05:23:13.404069 12223 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:13.404719 12223 heartbeater.cc:511] Master 127.8.124.254:34117 requested a full tablet report, sending...
I20260928 05:23:13.406193 11767 ts_manager.cc:194] Registered new tserver with Master: 6a7a5f612ad645b9b29a4b0330c5b26e (127.8.124.195:33667)
I20260928 05:23:13.406932 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:23:13.407088 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:43229
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:13.449105 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.436402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.8.124.254:34117)
0928 05:23:13.436551 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:13.436556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:13.437036 (+ 480us) server_negotiation.cc:407] Connection header received
0928 05:23:13.437230 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:13.437235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:13.437303 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:13.437402 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:13.438670 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.439385 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.440612 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.441002 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.441107 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.444524 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:13.444529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:13.444532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:13.444563 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:13.446828 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.447372 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.447376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.447378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.447493 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.447841 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.447845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.447849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.448132 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:13.448337 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:13.448628 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:23:13.448829 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 05:23:13.449568 11767 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35504
I20260928 05:23:13.451210 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11828
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:13.465307 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.452243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.8.124.193:35439)
0928 05:23:13.452676 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:13.452683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:13.452708 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:23:13.452775 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:13.452784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:13.452997 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:13.453139 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:13.454212 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.455149 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.456280 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.456822 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.456934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.460237 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:13.460247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:13.460261 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:13.460292 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:13.462696 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.463295 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.463301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.463303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.463483 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.463820 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.463825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.463829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.464156 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:13.464452 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:13.464703 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 05:23:13.464955 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":222,"threads_started":1}
W20260928 05:23:13.466709 11951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37506
W20260928 05:23:13.467753 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:13.467942 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11828
I20260928 05:23:13.480049 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11962
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:13.493873 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.480754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.8.124.194:45271)
0928 05:23:13.481210 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:13.481217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:13.481241 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:13.481295 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:13.481301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:13.481495 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:13.481634 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:13.482661 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.483588 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.484748 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.485261 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.485353 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.488986 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:13.488996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:13.489010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:13.489040 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:13.491675 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.492210 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.492216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.492218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.492325 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.492568 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.492570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.492572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.492872 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:13.493077 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:13.493261 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:23:13.493445 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":230,"threads_started":1}
W20260928 05:23:13.495028 12085 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34676
W20260928 05:23:13.495983 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11962: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:13.496102 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11962
I20260928 05:23:13.508018 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12095
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[11722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572993, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:13.522560 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:13.508744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.8.124.195:33667)
0928 05:23:13.509286 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:13.509293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:13.509321 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:23:13.509374 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:13.509380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:13.509570 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:13.509712 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:13.510855 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.511845 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.512943 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:13.513458 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:13.513547 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:13.517294 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:13.517305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:13.517318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:13.517348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:13.519816 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.520573 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.520582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.520586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.520722 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:13.521059 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:13.521064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:13.521068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:13.521382 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:13.521668 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:13.521979 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 05:23:13.522212 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":259,"threads_started":1}
W20260928 05:23:13.523789 12218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48516
W20260928 05:23:13.524771 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:23:13.524938 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12095
I20260928 05:23:13.536921 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 11735
I20260928 05:23:13.537905 11797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:13.635090 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 11735
2026-09-28T05:23:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3298 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[12240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[12240](info): set up 2 sockets
Sep 28 05:23:13 dist-test-slave-515x krb5kdc[12240](info): commencing operation
krb5kdc: starting...
W20260928 05:23:14.621701 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:14 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:14.792366 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36787
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:15.092026 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:15.092324 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:15.092449 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:15.099540 12253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:15.099658 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:15.099738 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:15.099818 12253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:15.099876 12253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:15.109964 12253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:15.111259 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:15.112529 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:15.123342 12258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:15.123374 12259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:15.123312 12261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:15.124264 12253 server_base.cc:1034] running on GCE node
I20260928 05:23:15.125029 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:15.126534 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:15.127734 12253 hybrid_clock.cc:648] HybridClock initialized: now 1790572995127683 us; error 42 us; skew 500 ppm
I20260928 05:23:15.128054 12253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:15 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572995, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:15.130982 12253 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:15.132313 12253 webserver.cc:460] Webserver started at http://127.8.124.254:40351/ using document root <none> and password file <none>
I20260928 05:23:15.132953 12253 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:15.133070 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:15.133368 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:15.136245 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "2703e25561b341f4bebde05ed8623e58"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "04fd7c6fc8415a7a7438fbc2aa355b1d"
server_key_iv: "9441cb5a1f5f89b48ae53881e4ed3c19"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.136972 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "2703e25561b341f4bebde05ed8623e58"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "04fd7c6fc8415a7a7438fbc2aa355b1d"
server_key_iv: "9441cb5a1f5f89b48ae53881e4ed3c19"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.141777 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 05:23:15.144888 12268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:15.146229 12253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:15.146384 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "2703e25561b341f4bebde05ed8623e58"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "04fd7c6fc8415a7a7438fbc2aa355b1d"
server_key_iv: "9441cb5a1f5f89b48ae53881e4ed3c19"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.146554 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:15.178622 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:15.183647 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:15.183938 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:15.200374 12253 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:36787
I20260928 05:23:15.200403 12319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:36787 every 8 connection(s)
I20260928 05:23:15.201680 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:15.205394 12320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:15.206877 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12253
I20260928 05:23:15.207090 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:15.207840 8691 external_mini_cluster.cc:1428] Setting key 2ed75645e26b70505e12d1e8801f7137
I20260928 05:23:15.218781 12320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58: Bootstrap starting.
Sep 28 05:23:15 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:15.222386 12320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:15.223497 12320 log.cc:824] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:15.226775 12320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58: No bootstrap required, opened a new log
I20260928 05:23:15.230733 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:15.209864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.8.124.254:36787)
0928 05:23:15.210625 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:15.210637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:15.210677 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:23:15.212210 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:15.212234 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:15.212614 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:15.212917 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:15.214591 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.215866 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.217569 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.218257 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.218384 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:15.222812 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:15.222820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:15.222835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:15.222871 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:15.226460 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:15.227343 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:15.227352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:15.227361 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:15.227581 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:15.228033 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:15.228038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:15.228041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:15.228602 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:15.228890 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:15.229302 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 05:23:15.229739 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":340,"threads_started":1}
I20260928 05:23:15.234584 12320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } }
I20260928 05:23:15.234999 12320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:15.235085 12320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703e25561b341f4bebde05ed8623e58, State: Initialized, Role: FOLLOWER
I20260928 05:23:15.235608 12320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } }
I20260928 05:23:15.235803 12320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:15.235918 12320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:15.236065 12320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:15.238065 12320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } }
I20260928 05:23:15.238489 12320 leader_election.cc:302] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2703e25561b341f4bebde05ed8623e58; no voters:
I20260928 05:23:15.239053 12320 leader_election.cc:288] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:15.239147 12325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:15.239880 12325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [term 1 LEADER]: Becoming Leader. State: Replica: 2703e25561b341f4bebde05ed8623e58, State: Running, Role: LEADER
I20260928 05:23:15.240339 12325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } }
I20260928 05:23:15.240856 12320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:15.242892 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2703e25561b341f4bebde05ed8623e58. Latest consensus state: current_term: 1 leader_uuid: "2703e25561b341f4bebde05ed8623e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } } }
I20260928 05:23:15.242949 12326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2703e25561b341f4bebde05ed8623e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2703e25561b341f4bebde05ed8623e58" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36787 } } }
I20260928 05:23:15.243286 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:15.243332 12326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:15.244474 12330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:15.249368 12330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:15.258126 12330 catalog_manager.cc:1347] Generated new cluster ID: 3367b4d564b344d08d9317389232578b
I20260928 05:23:15.258215 12330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:15.271921 12330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:15.273092 12330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:15.290267 12330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2703e25561b341f4bebde05ed8623e58: Generated new TSK 0
I20260928 05:23:15.291117 12330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:15.338853 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:15.650357 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:15.650602 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:15.650727 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:15.657836 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:15.657945 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:15.658094 12346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:15.667410 12346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:15.668586 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:15.669876 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:15.682053 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:15.682158 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:15.682126 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:15.682967 12346 server_base.cc:1034] running on GCE node
I20260928 05:23:15.683573 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:15.684453 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:15.685668 12346 hybrid_clock.cc:648] HybridClock initialized: now 1790572995685615 us; error 45 us; skew 500 ppm
I20260928 05:23:15.685958 12346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:15 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572995, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:15.688776 12346 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:15.690140 12346 webserver.cc:460] Webserver started at http://127.8.124.193:33201/ using document root <none> and password file <none>
I20260928 05:23:15.690795 12346 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:15.690907 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:15.691179 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:15.694018 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "4b811144b1e24f3fb32f5f887df8bfc7"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "f4d4aed040651bd1a37e68dbc0d1f8b1"
server_key_iv: "9186db04eb7bb7864022def434aac176"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.694763 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "4b811144b1e24f3fb32f5f887df8bfc7"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "f4d4aed040651bd1a37e68dbc0d1f8b1"
server_key_iv: "9186db04eb7bb7864022def434aac176"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.699592 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:23:15.702749 12361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:15.703933 12346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260928 05:23:15.704069 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "4b811144b1e24f3fb32f5f887df8bfc7"
format_stamp: "Formatted at 2026-09-28 05:23:15 on dist-test-slave-515x"
server_key: "f4d4aed040651bd1a37e68dbc0d1f8b1"
server_key_iv: "9186db04eb7bb7864022def434aac176"
server_key_version: "encryptionkey@0"
I20260928 05:23:15.704238 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:15.726713 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:15.730561 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:15.730804 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:15.731698 12346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:15.732954 12346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:15.733032 12346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:15.733142 12346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:15.733202 12346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:15.761574 12346 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:38163
I20260928 05:23:15.761639 12473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:38163 every 8 connection(s)
I20260928 05:23:15.762821 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:15.767051 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12346
I20260928 05:23:15.767248 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:15.768035 8691 external_mini_cluster.cc:1428] Setting key defe84fa6a4f31fb895442f1eafbd29b
Sep 28 05:23:15 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572995, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:15.782661 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:15.765120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:39829 (local address 127.8.124.254:36787)
0928 05:23:15.765579 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:15.765585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:15.766562 (+ 977us) server_negotiation.cc:407] Connection header received
0928 05:23:15.768934 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:15.768939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:15.769005 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:15.769110 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:15.770878 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.771578 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.773562 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.774008 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.774101 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:15.777661 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:15.777665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:15.777669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:15.777699 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:15.779883 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:15.780652 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:15.780655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:15.780658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:15.780755 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:15.781145 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:15.781148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:15.781150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:15.781383 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:15.781484 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:15.782273 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 05:23:15.782421 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260928 05:23:15.784099 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:15.765742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36787 (local address 127.8.124.193:39829)
0928 05:23:15.766399 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:15.766449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:15.768671 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:15.769301 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:15.769309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:15.769902 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:15.770686 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:15.770715 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.771720 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.771726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:15.773418 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:15.773433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:15.774203 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:15.774209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:15.774475 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:15.775147 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:15.775174 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:15.777432 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:15.780011 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:15.780021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:15.780038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:15.780565 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:15.780838 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:15.780842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:15.780846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:15.781057 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:15.781487 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:15.781494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:15.781922 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 05:23:15.782219 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 05:23:15.782715 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":245,"threads_started":1}
I20260928 05:23:15.785490 12474 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36787
I20260928 05:23:15.785823 12474 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:23:15.786556 12474 heartbeater.cc:511] Master 127.8.124.254:36787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:23:15.788486 12285 ts_manager.cc:194] Registered new tserver with Master: 4b811144b1e24f3fb32f5f887df8bfc7 (127.8.124.193:38163)
I20260928 05:23:15.790524 12285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:39829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:15.809135 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:16.106865 12480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:16.107167 12480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:16.107292 12480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:16.114523 12480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:16.114645 12480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:16.114796 12480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:16.124557 12480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:16.125854 12480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:16.127117 12480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:16.139113 12485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:16.139127 12486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:16.139077 12488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:16.139914 12480 server_base.cc:1034] running on GCE node
I20260928 05:23:16.140434 12480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:16.141292 12480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:16.142479 12480 hybrid_clock.cc:648] HybridClock initialized: now 1790572996142426 us; error 48 us; skew 500 ppm
I20260928 05:23:16.142759 12480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:16.145627 12480 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:16.146983 12480 webserver.cc:460] Webserver started at http://127.8.124.194:36421/ using document root <none> and password file <none>
I20260928 05:23:16.147606 12480 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:16.147717 12480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:16.147989 12480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:16.150878 12480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "4875304e0e034f99beffba00e897e7fe"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "f6b57f7a25842ff7361d2f79a2573e68"
server_key_iv: "a4b29b78ae8dd09c2006ac7133bbb8f1"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.151598 12480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "4875304e0e034f99beffba00e897e7fe"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "f6b57f7a25842ff7361d2f79a2573e68"
server_key_iv: "a4b29b78ae8dd09c2006ac7133bbb8f1"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.156330 12480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:23:16.159534 12495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.160748 12480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:16.160890 12480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "4875304e0e034f99beffba00e897e7fe"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "f6b57f7a25842ff7361d2f79a2573e68"
server_key_iv: "a4b29b78ae8dd09c2006ac7133bbb8f1"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.161057 12480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:16.198910 12480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:16.203920 12480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:16.204203 12480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:16.205181 12480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:16.214293 12480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:16.214385 12480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.214512 12480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:16.214561 12480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.243402 12480 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45649
I20260928 05:23:16.243443 12607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45649 every 8 connection(s)
I20260928 05:23:16.244608 12480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:16.246802 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12480
I20260928 05:23:16.246999 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:16.247435 8691 external_mini_cluster.cc:1428] Setting key dc9f55500fae05dd1c370553887d1442
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:16.264236 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.246935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:49385 (local address 127.8.124.254:36787)
0928 05:23:16.247138 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.247145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.248447 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 05:23:16.250343 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.250349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.250427 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.250507 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.252035 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.252741 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.254286 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.254659 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.254764 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.258331 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.258337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.258341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.258379 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.261283 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.261995 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.262001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.262004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.262156 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.262525 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.262531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.262534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.262875 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.263026 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.263669 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.263883 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:23:16.265213 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.247523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36787 (local address 127.8.124.194:49385)
0928 05:23:16.248286 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:16.248339 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:16.250072 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:16.250767 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:16.250779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:16.251327 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:16.251883 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:16.251899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.252880 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.252884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:16.254151 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:16.254162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.254871 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.254878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:16.255143 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.255780 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:16.255814 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:16.258123 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:16.261416 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:16.261422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:16.261435 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:16.261871 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:16.262248 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:16.262251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:16.262253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:16.262410 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:16.262978 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:16.262983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:16.263289 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 05:23:16.263547 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 05:23:16.264053 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":304,"threads_started":1}
I20260928 05:23:16.266496 12608 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36787
I20260928 05:23:16.266750 12608 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:16.267505 12608 heartbeater.cc:511] Master 127.8.124.254:36787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
I20260928 05:23:16.268872 12285 ts_manager.cc:194] Registered new tserver with Master: 4875304e0e034f99beffba00e897e7fe (127.8.124.194:45649)
I20260928 05:23:16.269696 12285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:49385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:16.288488 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:16.590691 12613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:16.590950 12613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:16.591080 12613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:16.597934 12613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:16.598034 12613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:16.598176 12613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:16.607393 12613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:16.608595 12613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:16.609840 12613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:16.621371 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:16.621466 12619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:16.621614 12621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:16.622426 12613 server_base.cc:1034] running on GCE node
I20260928 05:23:16.622906 12613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:16.623679 12613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:16.624872 12613 hybrid_clock.cc:648] HybridClock initialized: now 1790572996624819 us; error 46 us; skew 500 ppm
I20260928 05:23:16.625142 12613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:16.627888 12613 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:16.629175 12613 webserver.cc:460] Webserver started at http://127.8.124.195:38621/ using document root <none> and password file <none>
I20260928 05:23:16.629812 12613 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:16.629930 12613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:16.630194 12613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:16.632877 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "9db194ff4da04ababe56c9aa94147f79"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "a540d291b40698d8cacdd3d1899cfb83"
server_key_iv: "ffe0dac09cc32e79e9a39ce2d1f98740"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.633571 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "9db194ff4da04ababe56c9aa94147f79"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "a540d291b40698d8cacdd3d1899cfb83"
server_key_iv: "ffe0dac09cc32e79e9a39ce2d1f98740"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.638123 12613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:23:16.641149 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.642457 12613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:16.642587 12613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "9db194ff4da04ababe56c9aa94147f79"
format_stamp: "Formatted at 2026-09-28 05:23:16 on dist-test-slave-515x"
server_key: "a540d291b40698d8cacdd3d1899cfb83"
server_key_iv: "ffe0dac09cc32e79e9a39ce2d1f98740"
server_key_version: "encryptionkey@0"
I20260928 05:23:16.642747 12613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:16.660430 12613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:16.664098 12613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:16.664364 12613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:16.665249 12613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:16.666541 12613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:16.666615 12613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.666728 12613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:16.666778 12613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:16.695446 12613 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:46613
I20260928 05:23:16.695505 12740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:46613 every 8 connection(s)
I20260928 05:23:16.696652 12613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:16.702946 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12613
I20260928 05:23:16.703140 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:16.703671 8691 external_mini_cluster.cc:1428] Setting key 8f6af8bb9e2cb2f2e0e7f9fba3b6d1a9
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:16.715116 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.698979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:48259 (local address 127.8.124.254:36787)
0928 05:23:16.699186 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.699192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.700325 (+ 1133us) server_negotiation.cc:407] Connection header received
0928 05:23:16.702008 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.702013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.702080 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.702164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.703984 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.704848 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.706708 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.707156 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.707253 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.710496 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.710501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.710504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.710534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.712672 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.713276 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.713280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.713282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.713382 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.713718 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.713721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.713723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.713984 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.714093 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.714698 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.714877 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:23:16.715898 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.699492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36787 (local address 127.8.124.195:48259)
0928 05:23:16.700149 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:16.700184 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:16.701764 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:16.702350 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:16.702358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:16.703025 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:16.703787 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:16.703813 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.705035 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.705040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:16.706580 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:16.706594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.707331 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.707336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:16.707537 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.708163 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:16.708188 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:16.710313 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:16.712782 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:16.712788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:16.712800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:16.713191 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:16.713453 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:16.713456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:16.713458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:16.713645 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:16.714071 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:16.714076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:16.714348 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 05:23:16.714558 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 05:23:16.714897 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":245,"threads_started":1}
I20260928 05:23:16.717094 12741 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36787
I20260928 05:23:16.717379 12741 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:16.718230 12741 heartbeater.cc:511] Master 127.8.124.254:36787 requested a full tablet report, sending...
I20260928 05:23:16.719523 12285 ts_manager.cc:194] Registered new tserver with Master: 9db194ff4da04ababe56c9aa94147f79 (127.8.124.195:46613)
I20260928 05:23:16.720289 12285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:48259
I20260928 05:23:16.730508 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:16.773557 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.759039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.8.124.193:38163)
0928 05:23:16.759660 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.759667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.759692 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:23:16.759754 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.759760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.759967 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.760091 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.761340 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.762885 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.764034 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.764851 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.764998 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.768436 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.768445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.768457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.768486 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.770973 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.771541 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.771549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.771552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.771684 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.772105 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.772109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.772112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.772480 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.772693 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.772972 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.773206 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":398,"threads_started":1}
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:16.790012 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.777484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.8.124.254:36787)
0928 05:23:16.777698 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.777704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.777871 (+ 167us) server_negotiation.cc:407] Connection header received
0928 05:23:16.778017 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.778021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.778094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.778173 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.779346 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.780059 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.781231 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.781681 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.781785 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.785330 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.785336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.785339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.785369 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.787751 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.788301 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.788306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.788308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.788419 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.788740 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.788745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.788749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.789029 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.789255 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.789510 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.789726 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 05:23:16.790617 12285 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55802
I20260928 05:23:16.791568 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12346
I20260928 05:23:16.793238 12474 heartbeater.cc:503] Master 127.8.124.254:36787 was elected leader, sending a full tablet report...
I20260928 05:23:16.803186 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.792381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.8.124.193:38163)
0928 05:23:16.792573 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.792579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.792694 (+ 115us) server_negotiation.cc:407] Connection header received
0928 05:23:16.792858 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.792863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.792948 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.793026 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.794335 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.795149 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.796326 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.796714 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.796802 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.798801 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.798806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.798809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.798839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.801096 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.801693 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.801697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.801700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.801812 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.802096 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.802099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.802101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.802324 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.802494 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.802727 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.802883 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:16.803699 12469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:16.883430 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12346
I20260928 05:23:16.895728 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12480
Sep 28 05:23:16 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:16.910085 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:16.896540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.8.124.194:45649)
0928 05:23:16.897016 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:16.897023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:16.897047 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:16.897107 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:16.897113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:16.897306 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:16.897425 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:16.898583 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.899549 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.900784 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:16.901286 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:16.901370 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:16.905056 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:16.905066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:16.905080 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:16.905111 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:16.907734 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.908322 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.908328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.908331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.908441 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:16.908757 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:16.908759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:16.908761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:16.909071 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:16.909270 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:16.909554 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 05:23:16.909775 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260928 05:23:16.911283 12603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:16.992532 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12480
I20260928 05:23:17.004552 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12613
Sep 28 05:23:17 dist-test-slave-515x krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:17.019876 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:17.005409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.8.124.195:46613)
0928 05:23:17.006027 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:17.006034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:17.006062 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:23:17.006119 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:17.006125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:17.006329 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:17.006470 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:17.007730 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:17.008738 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:17.010085 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:17.010656 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:17.010766 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:17.014885 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:17.014897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:17.014912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:17.014946 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:17.017650 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:17.018251 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:17.018257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:17.018260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:17.018405 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:17.018668 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:17.018671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:17.018673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:17.018979 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:17.019201 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:17.019430 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 05:23:17.019610 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
I20260928 05:23:17.021128 12736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:17.101106 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12613
I20260928 05:23:17.113181 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12253
I20260928 05:23:17.114328 12315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:17.215127 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12253
2026-09-28T05:23:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3579 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:17 dist-test-slave-515x krb5kdc[12760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:17 dist-test-slave-515x krb5kdc[12760](info): set up 2 sockets
Sep 28 05:23:17 dist-test-slave-515x krb5kdc[12760](info): commencing operation
krb5kdc: starting...
W20260928 05:23:18.201742 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:18 dist-test-slave-515x krb5kdc[12760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:18.376255 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40787
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:18.674674 12773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:18.674939 12773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:18.675067 12773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:18.682052 12773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:18.682152 12773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:18.682219 12773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:18.682291 12773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:18.682356 12773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:18.691941 12773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:18.693218 12773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:18.694576 12773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:18.705068 12778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:18.705107 12779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:18.705252 12781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:18.705991 12773 server_base.cc:1034] running on GCE node
I20260928 05:23:18.706663 12773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:18.707908 12773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:18.709102 12773 hybrid_clock.cc:648] HybridClock initialized: now 1790572998709057 us; error 40 us; skew 500 ppm
I20260928 05:23:18.709406 12773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:18 dist-test-slave-515x krb5kdc[12760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:18.715476 12773 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:18.716524 12773 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:18.716619 12773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:18.716925 12773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:18.719779 12773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "081b4abf4748406ea70a26286afbfa37"
format_stamp: "Formatted at 2026-09-28 05:23:18 on dist-test-slave-515x"
server_key: "32dc9b75a51c3bf0996dba3ff8517815"
server_key_iv: "0591372c04436031e45e4e92d90bc971"
server_key_version: "encryptionkey@0"
I20260928 05:23:18.720508 12773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "081b4abf4748406ea70a26286afbfa37"
format_stamp: "Formatted at 2026-09-28 05:23:18 on dist-test-slave-515x"
server_key: "32dc9b75a51c3bf0996dba3ff8517815"
server_key_iv: "0591372c04436031e45e4e92d90bc971"
server_key_version: "encryptionkey@0"
I20260928 05:23:18.725343 12773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 05:23:18.728641 12787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:18.729871 12773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:18.730055 12773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "081b4abf4748406ea70a26286afbfa37"
format_stamp: "Formatted at 2026-09-28 05:23:18 on dist-test-slave-515x"
server_key: "32dc9b75a51c3bf0996dba3ff8517815"
server_key_iv: "0591372c04436031e45e4e92d90bc971"
server_key_version: "encryptionkey@0"
I20260928 05:23:18.730307 12773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:18.751238 12773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:18.754877 12773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:18.755106 12773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:18.770550 12773 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:40787
I20260928 05:23:18.770569 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:40787 every 8 connection(s)
I20260928 05:23:18.771389 12773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:18.772099 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12773
I20260928 05:23:18.772332 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:18.773052 8691 external_mini_cluster.cc:1428] Setting key 18f6b15f8f3611dab3479015d27b523f
I20260928 05:23:18.775717 12839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:23:18 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:18.787530 12839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: Bootstrap starting.
I20260928 05:23:18.791365 12839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:18.792477 12839 log.cc:824] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:18.795503 12839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: No bootstrap required, opened a new log
I20260928 05:23:18.795543 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:18.775072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.8.124.254:40787)
0928 05:23:18.775786 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:18.775797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:18.775852 (+ 55us) server_negotiation.cc:407] Connection header received
0928 05:23:18.777384 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:18.777403 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:18.777804 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:18.778127 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:18.779769 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:18.781069 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:18.782851 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:18.783490 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:18.783620 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:18.787770 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:18.787780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:18.787798 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:18.787845 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:18.791323 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:18.792014 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:18.792022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:18.792030 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:18.792225 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:18.792541 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:18.792545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:18.792548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:18.793223 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:18.793442 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:18.793948 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 05:23:18.794349 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":335,"threads_started":1}
I20260928 05:23:18.803166 12839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } }
I20260928 05:23:18.803516 12839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:18.803586 12839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081b4abf4748406ea70a26286afbfa37, State: Initialized, Role: FOLLOWER
I20260928 05:23:18.804108 12839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } }
I20260928 05:23:18.804311 12839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:18.804405 12839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:18.804518 12839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:18.806587 12839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } }
I20260928 05:23:18.807080 12839 leader_election.cc:302] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 081b4abf4748406ea70a26286afbfa37; no voters:
I20260928 05:23:18.807664 12839 leader_election.cc:288] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:18.807799 12844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:18.808403 12844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [term 1 LEADER]: Becoming Leader. State: Replica: 081b4abf4748406ea70a26286afbfa37, State: Running, Role: LEADER
I20260928 05:23:18.808943 12844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } }
I20260928 05:23:18.809588 12839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:18.811532 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "081b4abf4748406ea70a26286afbfa37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } } }
I20260928 05:23:18.811641 12846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 081b4abf4748406ea70a26286afbfa37. Latest consensus state: current_term: 1 leader_uuid: "081b4abf4748406ea70a26286afbfa37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b4abf4748406ea70a26286afbfa37" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40787 } } }
I20260928 05:23:18.811906 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:18.811960 12846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:18.813725 12852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:18.817760 12852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:18.825085 12852 catalog_manager.cc:1347] Generated new cluster ID: 12e23bbbf1814ff19612da03ffffc382
I20260928 05:23:18.825187 12852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:18.848083 12852 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:18.849897 12852 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:18.873416 12852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: Generated new TSK 0
I20260928 05:23:18.874482 12852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:18.915239 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:19.219758 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:19.220016 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:19.220139 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:19.227064 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:19.227162 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:19.227303 12865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:19.236847 12865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:19.238117 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:19.239534 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:19.251830 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:19.251865 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:19.251890 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:19.252494 12865 server_base.cc:1034] running on GCE node
I20260928 05:23:19.253039 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:19.253968 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:19.255151 12865 hybrid_clock.cc:648] HybridClock initialized: now 1790572999255104 us; error 46 us; skew 500 ppm
I20260928 05:23:19.255435 12865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:19 dist-test-slave-515x krb5kdc[12760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:19.258442 12865 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:19.259438 12865 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:19.259524 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:19.259814 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:19.262632 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "9f794128c4aa910be96ea7a59bceadaf"
server_key_iv: "afe50f28392b7afd686f6242ab039a4f"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.263379 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "9f794128c4aa910be96ea7a59bceadaf"
server_key_iv: "afe50f28392b7afd686f6242ab039a4f"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.268146 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:19.271339 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.272548 12865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:23:19.272682 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "9f794128c4aa910be96ea7a59bceadaf"
server_key_iv: "afe50f28392b7afd686f6242ab039a4f"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.272877 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:19.293825 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:19.297428 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:19.297675 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:19.298491 12865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:19.299691 12865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:19.299782 12865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.299880 12865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:19.299947 12865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.327531 12865 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:40867
I20260928 05:23:19.327582 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:40867 every 8 connection(s)
I20260928 05:23:19.328286 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:19.331341 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12865
I20260928 05:23:19.331511 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:19.332091 8691 external_mini_cluster.cc:1428] Setting key b5536b02ee80bb21c3448d8fb1e48785
Sep 28 05:23:19 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:19.348390 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:19.330639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:41701 (local address 127.8.124.254:40787)
0928 05:23:19.331071 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:19.331077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:19.332157 (+ 1080us) server_negotiation.cc:407] Connection header received
0928 05:23:19.334197 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:19.334201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:19.334278 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:19.334399 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:19.335807 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.336726 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.338275 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.338781 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.338876 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:19.342524 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:19.342530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:19.342535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:19.342572 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:19.345409 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:19.346133 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:19.346139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:19.346143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:19.346275 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:19.346664 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:19.346669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:19.346673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:19.347039 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:19.347182 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:19.347810 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 05:23:19.348035 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":201,"threads_started":1}
I20260928 05:23:19.349092 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:19.331224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40787 (local address 127.8.124.193:41701)
0928 05:23:19.331995 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:19.332040 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:19.333945 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:19.334589 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:19.334598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:19.335106 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:19.335655 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:19.335672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.336872 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.336877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:19.338156 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:19.338167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.338996 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.339003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:19.339334 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:19.340040 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:19.340062 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:19.342316 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:19.345547 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:19.345553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:19.345565 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:19.346011 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:19.346386 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:19.346389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:19.346392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:19.346556 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:19.347164 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:19.347170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:19.347474 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 05:23:19.347675 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 05:23:19.348038 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":316,"threads_started":1}
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:23:19.350610 12992 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40787
I20260928 05:23:19.350880 12992 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:19.351539 12992 heartbeater.cc:511] Master 127.8.124.254:40787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:23:19.353561 12804 ts_manager.cc:194] Registered new tserver with Master: 59342cfdba4c4c4c8ffdef7a0ff93ce8 (127.8.124.193:40867)
I20260928 05:23:19.354998 12804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:41701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:19.374387 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:19.677042 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:19.677320 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:19.677449 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:19.684617 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:19.684719 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:19.684844 12998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:19.694437 12998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:19.695665 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:19.696874 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:19.708844 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:19.708823 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:19.708923 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:19.709589 12998 server_base.cc:1034] running on GCE node
I20260928 05:23:19.710225 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:19.711163 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:19.712370 12998 hybrid_clock.cc:648] HybridClock initialized: now 1790572999712322 us; error 48 us; skew 500 ppm
I20260928 05:23:19.712662 12998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:19 dist-test-slave-515x krb5kdc[12760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:19.715631 12998 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:19.716648 12998 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:19.716743 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:19.717023 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:19.719774 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "75599bc96dab48c18ff95930268f81ad"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "76ee377bd0b20e49246ed3a359b1d6aa"
server_key_iv: "96e38641c562b338b605a12bda448944"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.720525 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "75599bc96dab48c18ff95930268f81ad"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "76ee377bd0b20e49246ed3a359b1d6aa"
server_key_iv: "96e38641c562b338b605a12bda448944"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.725174 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 05:23:19.728404 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.729640 12998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:19.729785 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "75599bc96dab48c18ff95930268f81ad"
format_stamp: "Formatted at 2026-09-28 05:23:19 on dist-test-slave-515x"
server_key: "76ee377bd0b20e49246ed3a359b1d6aa"
server_key_iv: "96e38641c562b338b605a12bda448944"
server_key_version: "encryptionkey@0"
I20260928 05:23:19.729966 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:19.744068 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:19.747650 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:19.747889 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:19.748733 12998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:19.750027 12998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:19.750121 12998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.750221 12998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:19.750304 12998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:19.779012 12998 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:37097
I20260928 05:23:19.779047 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:37097 every 8 connection(s)
I20260928 05:23:19.779793 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:19.780629 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 12998
I20260928 05:23:19.780836 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:19.781402 8691 external_mini_cluster.cc:1428] Setting key 5cc41d51fa9824630e44f989739bfc80
Sep 28 05:23:19 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572999, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
I20260928 05:23:19.805557 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:19.782625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:44269 (local address 127.8.124.254:40787)
0928 05:23:19.783027 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:19.783033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:19.789191 (+ 6158us) server_negotiation.cc:407] Connection header received
0928 05:23:19.791373 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:19.791379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:19.791456 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:19.791538 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:19.792950 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.794031 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.795621 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.796209 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.796322 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:19.799591 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:19.799598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:19.799603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:19.799644 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:19.802726 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:19.803460 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:19.803465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:19.803469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:19.803606 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:19.804012 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:19.804015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:19.804017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:19.804257 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:19.804354 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:19.805053 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 05:23:19.805266 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260928 05:23:19.806775 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:19.782856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40787 (local address 127.8.124.194:44269)
0928 05:23:19.789047 (+ 6191us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:19.789093 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:19.791113 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:19.791769 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:19.791776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:19.792261 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:19.792796 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:19.792812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.794191 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.794196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:19.795488 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:19.795500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:19.796429 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:19.796434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:19.796624 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:19.797244 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:19.797282 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:19.799372 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:19.802856 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:19.802864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:19.802890 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:19.803365 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:19.803732 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:19.803736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:19.803739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:19.803924 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:19.804379 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:19.804384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:19.804715 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 05:23:19.804957 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 05:23:19.805385 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5922,"thread_start_us":5697,"threads_started":1}
I20260928 05:23:19.808238 13125 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40787
I20260928 05:23:19.808511 13125 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:19.809135 13125 heartbeater.cc:511] Master 127.8.124.254:40787 requested a full tablet report, sending...
I20260928 05:23:19.810563 12804 ts_manager.cc:194] Registered new tserver with Master: 75599bc96dab48c18ff95930268f81ad (127.8.124.194:37097)
I20260928 05:23:19.811352 12804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:44269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:19.828910 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:20.129588 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:20.129879 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:20.130013 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:20.137226 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:20.137349 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:20.137501 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:20.146875 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:20.148082 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:20.149338 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:20.160916 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:20.161056 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:20.161092 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:20.161844 13130 server_base.cc:1034] running on GCE node
I20260928 05:23:20.162387 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:20.163236 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:20.164429 13130 hybrid_clock.cc:648] HybridClock initialized: now 1790573000164379 us; error 50 us; skew 500 ppm
I20260928 05:23:20.164693 13130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:20 dist-test-slave-515x krb5kdc[12760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573000, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:20.167446 13130 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:20.168466 13130 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:20.168561 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:20.168835 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:20.171569 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "34b7e42371a445c1b7f344134bc6e6c0"
format_stamp: "Formatted at 2026-09-28 05:23:20 on dist-test-slave-515x"
server_key: "46926a8d37faf06493b0f331c7e4f619"
server_key_iv: "e9e9f81d9a0e3982cddf0b06d4cf3dd6"
server_key_version: "encryptionkey@0"
I20260928 05:23:20.172293 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "34b7e42371a445c1b7f344134bc6e6c0"
format_stamp: "Formatted at 2026-09-28 05:23:20 on dist-test-slave-515x"
server_key: "46926a8d37faf06493b0f331c7e4f619"
server_key_iv: "e9e9f81d9a0e3982cddf0b06d4cf3dd6"
server_key_version: "encryptionkey@0"
I20260928 05:23:20.177309 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:23:20.180528 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:20.181792 13130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:20.181924 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "34b7e42371a445c1b7f344134bc6e6c0"
format_stamp: "Formatted at 2026-09-28 05:23:20 on dist-test-slave-515x"
server_key: "46926a8d37faf06493b0f331c7e4f619"
server_key_iv: "e9e9f81d9a0e3982cddf0b06d4cf3dd6"
server_key_version: "encryptionkey@0"
I20260928 05:23:20.182101 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:20.194624 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:20.198580 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:20.198823 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:20.199708 13130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:20.200979 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:20.201067 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:20.201169 13130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:20.201236 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:20.229970 13130 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:38117
I20260928 05:23:20.230032 13256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:38117 every 8 connection(s)
I20260928 05:23:20.230803 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:20.234680 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13130
I20260928 05:23:20.234865 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:20.235252 8691 external_mini_cluster.cc:1428] Setting key 6cb840a71dd0da4eb99ad91bedcedc33
Sep 28 05:23:20 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573000, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:20.250162 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.233017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:55625 (local address 127.8.124.254:40787)
0928 05:23:20.233182 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.233188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.234554 (+ 1366us) server_negotiation.cc:407] Connection header received
0928 05:23:20.236453 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.236458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.236539 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.236633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:20.238156 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.238880 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.240326 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.240709 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.240807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.244460 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:20.244466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:20.244471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:20.244510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:20.247199 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.247848 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.247852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.247854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.247964 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.248420 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.248424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.248427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.248746 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:20.248867 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.249584 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.249816 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:23:20.251068 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.233527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40787 (local address 127.8.124.195:55625)
0928 05:23:20.234384 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.234440 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.236203 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.236810 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.236820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.237343 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:20.238011 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.238034 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.239005 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.239009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.240194 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.240205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.240890 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.240897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.241090 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.241678 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:20.241704 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:20.244243 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:20.247325 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:20.247332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:20.247344 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:20.247745 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:20.248066 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:20.248071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:20.248074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:20.248287 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:20.248867 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:20.248875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:20.249240 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.249455 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.249940 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":359,"threads_started":1}
I20260928 05:23:20.252341 13257 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40787
I20260928 05:23:20.252620 13257 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:20.253230 13257 heartbeater.cc:511] Master 127.8.124.254:40787 requested a full tablet report, sending...
I20260928 05:23:20.254534 12804 ts_manager.cc:194] Registered new tserver with Master: 34b7e42371a445c1b7f344134bc6e6c0 (127.8.124.195:38117)
I20260928 05:23:20.255290 12804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:55625
I20260928 05:23:20.262240 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:23:20.277814 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.266365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.8.124.254:40787)
0928 05:23:20.266558 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.266564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.266887 (+ 323us) server_negotiation.cc:407] Connection header received
0928 05:23:20.267015 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.267018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.267074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.267146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:20.268438 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.269115 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.270380 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.270812 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.270907 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.273093 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:20.273098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:20.273101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:20.273129 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:20.275484 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.276096 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.276100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.276102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.276216 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.276555 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.276557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.276560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.276853 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:20.277040 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.277369 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.277509 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:23:20.281329 12804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:20.287956 12804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:20.308961 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.298839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38117 (local address 127.0.0.1:33816)
0928 05:23:20.299530 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.299567 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.299736 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.301825 (+ 2089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.301833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.301864 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:20.302277 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.302289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.304128 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.304133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.306403 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.306414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.307974 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.307981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.308158 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.308210 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.308320 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.308496 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":9,"thread_start_us":287,"threads_started":1}
I20260928 05:23:20.309743 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.299155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.8.124.195:38117)
0928 05:23:20.301299 (+ 2144us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.301305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.301329 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:20.301403 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.301408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.301579 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.302465 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:20.302724 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.304007 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.306549 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.307797 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.307932 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.309184 (+ 1252us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.309254 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.309428 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2015,"thread_start_us":1863,"threads_started":1}
I20260928 05:23:20.310009 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.299742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.8.124.193:40867)
0928 05:23:20.300253 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.300263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.300774 (+ 511us) server_negotiation.cc:407] Connection header received
0928 05:23:20.301784 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.301793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.302047 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.302226 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:20.303073 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.304243 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.307049 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.308395 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.308513 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.308693 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.309537 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.309721 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260928 05:23:20.310376 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.299536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:40867 (local address 127.0.0.1:57920)
0928 05:23:20.300650 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.300667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.300803 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.302477 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.302481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.302504 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:20.302900 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.302912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.304377 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.304382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.306902 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.306916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.308843 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.308849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.309019 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.309050 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.309142 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.309267 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":71,"thread_start_us":865,"threads_started":1}
I20260928 05:23:20.311311 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.300536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.8.124.194:37097)
0928 05:23:20.301122 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.301131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.302383 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 05:23:20.302480 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.302487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.302676 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.302841 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:20.303594 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.304925 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.307328 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.308513 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.308629 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.308796 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.309537 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.309708 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":338,"threads_started":1}
I20260928 05:23:20.311721 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.299944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:37097 (local address 127.0.0.1:50618)
0928 05:23:20.302103 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.302120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.302239 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.303073 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.303077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.303099 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:20.303429 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.303440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.305090 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.305095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.307177 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.307186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.308930 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.308937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.309094 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.309124 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.309194 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.309313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"mutex_wait_us":16,"thread_start_us":1901,"threads_started":1}
I20260928 05:23:20.313432 13192 tablet_service.cc:1467] Processing CreateTablet for tablet ad685a5911494641b8e48dfbe26bda46 (DEFAULT_TABLE table=test-table [id=4e567fd6653641ae92f127a0c5d390d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:20.313453 12927 tablet_service.cc:1467] Processing CreateTablet for tablet ad685a5911494641b8e48dfbe26bda46 (DEFAULT_TABLE table=test-table [id=4e567fd6653641ae92f127a0c5d390d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:20.314815 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad685a5911494641b8e48dfbe26bda46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:20.314821 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad685a5911494641b8e48dfbe26bda46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:20.314693 13060 tablet_service.cc:1467] Processing CreateTablet for tablet ad685a5911494641b8e48dfbe26bda46 (DEFAULT_TABLE table=test-table [id=4e567fd6653641ae92f127a0c5d390d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:20.315699 13060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad685a5911494641b8e48dfbe26bda46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:20.326210 13275 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Bootstrap starting.
I20260928 05:23:20.328449 13276 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Bootstrap starting.
I20260928 05:23:20.329807 13275 tablet_bootstrap.cc:654] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:20.331101 13275 log.cc:824] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:20.332001 13276 tablet_bootstrap.cc:654] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:20.333269 13276 log.cc:824] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:20.334564 13275 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: No bootstrap required, opened a new log
I20260928 05:23:20.334870 13275 ts_tablet_manager.cc:1397] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260928 05:23:20.335924 13278 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Bootstrap starting.
I20260928 05:23:20.336238 13276 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: No bootstrap required, opened a new log
I20260928 05:23:20.336503 13276 ts_tablet_manager.cc:1397] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:23:20.338579 13278 tablet_bootstrap.cc:654] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:20.339730 13278 log.cc:824] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:20.341982 13278 tablet_bootstrap.cc:492] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: No bootstrap required, opened a new log
I20260928 05:23:20.342262 13278 ts_tablet_manager.cc:1397] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 05:23:20.345043 13275 raft_consensus.cc:348] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.345446 13275 raft_consensus.cc:374] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:20.345542 13275 raft_consensus.cc:729] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75599bc96dab48c18ff95930268f81ad, State: Initialized, Role: FOLLOWER
I20260928 05:23:20.346158 13275 consensus_queue.cc:260] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.346207 13276 raft_consensus.cc:348] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.346588 13276 raft_consensus.cc:374] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:20.346752 13276 raft_consensus.cc:729] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b7e42371a445c1b7f344134bc6e6c0, State: Initialized, Role: FOLLOWER
I20260928 05:23:20.347172 13125 heartbeater.cc:503] Master 127.8.124.254:40787 was elected leader, sending a full tablet report...
I20260928 05:23:20.347433 13276 consensus_queue.cc:260] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.347836 13275 ts_tablet_manager.cc:1428] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260928 05:23:20.348417 13257 heartbeater.cc:503] Master 127.8.124.254:40787 was elected leader, sending a full tablet report...
I20260928 05:23:20.348933 13276 ts_tablet_manager.cc:1428] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 05:23:20.351202 13278 raft_consensus.cc:348] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.351533 13278 raft_consensus.cc:374] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:20.351622 13278 raft_consensus.cc:729] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59342cfdba4c4c4c8ffdef7a0ff93ce8, State: Initialized, Role: FOLLOWER
I20260928 05:23:20.352160 13278 consensus_queue.cc:260] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.353236 12992 heartbeater.cc:503] Master 127.8.124.254:40787 was elected leader, sending a full tablet report...
I20260928 05:23:20.353631 13278 ts_tablet_manager.cc:1428] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260928 05:23:20.355609 13282 raft_consensus.cc:482] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:20.355858 13282 raft_consensus.cc:504] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.357470 13282 leader_election.cc:288] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75599bc96dab48c18ff95930268f81ad (127.8.124.194:37097), 59342cfdba4c4c4c8ffdef7a0ff93ce8 (127.8.124.193:40867)
I20260928 05:23:20.363112 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.357853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:36795 (local address 127.8.124.193:40867)
0928 05:23:20.358027 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.358034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.358081 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:23:20.358298 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.358302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.358367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.358455 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:20.358797 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.359693 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.361363 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.362311 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.362412 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.362459 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.362788 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.362883 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:23:20.363121 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.357712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:40867 (local address 127.8.124.195:36795)
0928 05:23:20.358007 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.358027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.358158 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.358488 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.358491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.358511 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:20.358675 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.358685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.359807 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.359813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.361218 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.361232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.362446 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.362453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.362647 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.362682 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.362778 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.362915 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 05:23:20.363837 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.357921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:37925 (local address 127.8.124.194:37097)
0928 05:23:20.358068 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.358072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.358564 (+ 492us) server_negotiation.cc:407] Connection header received
0928 05:23:20.358743 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.358748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.358814 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.358873 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:20.359347 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.360132 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.361880 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.362778 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.362901 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.362960 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.363537 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.363656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:20.364121 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad685a5911494641b8e48dfbe26bda46" candidate_uuid: "34b7e42371a445c1b7f344134bc6e6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" is_pre_election: true
I20260928 05:23:20.364472 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.357827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:37097 (local address 127.8.124.195:37925)
0928 05:23:20.358472 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:20.358489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:20.358636 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:20.358901 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:20.358905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:20.358927 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:20.359230 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.359240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.360266 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.360270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.361750 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:20.361763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.363189 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.363199 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:20.363365 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.363398 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:23:20.364054 (+ 656us) client_negotiation.cc:240] Negotiation successful
0928 05:23:20.364168 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"spinlock_wait_cycles":16640,"thread_start_us":369,"threads_started":1}
I20260928 05:23:20.364634 12947 raft_consensus.cc:2454] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34b7e42371a445c1b7f344134bc6e6c0 in term 0.
I20260928 05:23:20.365190 13146 leader_election.cc:302] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b7e42371a445c1b7f344134bc6e6c0, 59342cfdba4c4c4c8ffdef7a0ff93ce8; no voters:
I20260928 05:23:20.365250 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad685a5911494641b8e48dfbe26bda46" candidate_uuid: "34b7e42371a445c1b7f344134bc6e6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75599bc96dab48c18ff95930268f81ad" is_pre_election: true
I20260928 05:23:20.365499 13282 raft_consensus.cc:2781] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:20.365674 13282 raft_consensus.cc:482] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:20.365778 13282 raft_consensus.cc:3037] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:20.365694 13080 raft_consensus.cc:2454] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34b7e42371a445c1b7f344134bc6e6c0 in term 0.
I20260928 05:23:20.367872 13282 raft_consensus.cc:504] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.368371 13282 leader_election.cc:288] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [CANDIDATE]: Term 1 election: Requested vote from peers 75599bc96dab48c18ff95930268f81ad (127.8.124.194:37097), 59342cfdba4c4c4c8ffdef7a0ff93ce8 (127.8.124.193:40867)
I20260928 05:23:20.368687 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad685a5911494641b8e48dfbe26bda46" candidate_uuid: "34b7e42371a445c1b7f344134bc6e6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75599bc96dab48c18ff95930268f81ad"
I20260928 05:23:20.368783 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad685a5911494641b8e48dfbe26bda46" candidate_uuid: "34b7e42371a445c1b7f344134bc6e6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8"
I20260928 05:23:20.368839 13080 raft_consensus.cc:3037] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:20.368959 12947 raft_consensus.cc:3037] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:20.370944 13080 raft_consensus.cc:2454] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34b7e42371a445c1b7f344134bc6e6c0 in term 1.
I20260928 05:23:20.371027 12947 raft_consensus.cc:2454] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34b7e42371a445c1b7f344134bc6e6c0 in term 1.
I20260928 05:23:20.371273 13147 leader_election.cc:302] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b7e42371a445c1b7f344134bc6e6c0, 75599bc96dab48c18ff95930268f81ad; no voters:
I20260928 05:23:20.371562 13282 raft_consensus.cc:2781] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:20.371941 13282 raft_consensus.cc:686] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 1 LEADER]: Becoming Leader. State: Replica: 34b7e42371a445c1b7f344134bc6e6c0, State: Running, Role: LEADER
I20260928 05:23:20.372365 13282 consensus_queue.cc:237] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } }
I20260928 05:23:20.376655 12803 catalog_manager.cc:5495] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34b7e42371a445c1b7f344134bc6e6c0 (127.8.124.195). New cstate: current_term: 1 leader_uuid: "34b7e42371a445c1b7f344134bc6e6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b7e42371a445c1b7f344134bc6e6c0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:20.390780 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.385850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.8.124.195:38117)
0928 05:23:20.386175 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.386180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.386198 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:20.386268 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.386272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.386372 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.386480 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:20.386844 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.387537 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.388826 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.389201 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.389302 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.389758 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:20.390216 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:20.390383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.390480 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.390556 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260928 05:23:20.398067 12947 raft_consensus.cc:1261] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 1 FOLLOWER]: Refusing update from remote peer 34b7e42371a445c1b7f344134bc6e6c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:20.398066 13080 raft_consensus.cc:1261] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 1 FOLLOWER]: Refusing update from remote peer 34b7e42371a445c1b7f344134bc6e6c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:20.399066 13285 consensus_queue.cc:1035] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75599bc96dab48c18ff95930268f81ad" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:20.399385 13282 consensus_queue.cc:1035] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59342cfdba4c4c4c8ffdef7a0ff93ce8" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:20.408381 13289 mvcc.cc:205] Tried to move back new op lower bound from 7334187009621815296 to 7334187009528446976. Current Snapshot: MvccSnapshot[applied={T|T < 7334187009621815296}]
I20260928 05:23:20.409279 13288 mvcc.cc:205] Tried to move back new op lower bound from 7334187009621815296 to 7334187009528446976. Current Snapshot: MvccSnapshot[applied={T|T < 7334187009621815296}]
I20260928 05:23:20.415609 13290 mvcc.cc:205] Tried to move back new op lower bound from 7334187009621815296 to 7334187009528446976. Current Snapshot: MvccSnapshot[applied={T|T < 7334187009621815296}]
I20260928 05:23:20.425314 12803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:20.425576 12803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:20.428917 12803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 081b4abf4748406ea70a26286afbfa37: Sending DeleteTablet for 3 replicas of tablet ad685a5911494641b8e48dfbe26bda46
I20260928 05:23:20.430066 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet ad685a5911494641b8e48dfbe26bda46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:20 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:57920
I20260928 05:23:20.430073 13192 tablet_service.cc:1514] Processing DeleteTablet for tablet ad685a5911494641b8e48dfbe26bda46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:20 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:33816
I20260928 05:23:20.430078 13060 tablet_service.cc:1514] Processing DeleteTablet for tablet ad685a5911494641b8e48dfbe26bda46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:20 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:50618
I20260928 05:23:20.432075 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12865
I20260928 05:23:20.441308 13299 tablet_replica.cc:330] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: stopping tablet replica
I20260928 05:23:20.442324 13299 raft_consensus.cc:2229] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:20.443557 13299 raft_consensus.cc:2258] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:20.446520 13299 ts_tablet_manager.cc:1905] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:20.453711 13298 tablet_replica.cc:330] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: stopping tablet replica
I20260928 05:23:20.455127 13300 tablet_replica.cc:330] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: stopping tablet replica
I20260928 05:23:20.455333 13299 ts_tablet_manager.cc:1918] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:20.455543 13299 log.cc:1196] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/ad685a5911494641b8e48dfbe26bda46
I20260928 05:23:20.455557 13298 raft_consensus.cc:2229] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:20.455565 13300 raft_consensus.cc:2229] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:20.456032 13300 raft_consensus.cc:2258] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:20.456133 13299 ts_tablet_manager.cc:1939] T ad685a5911494641b8e48dfbe26bda46 P 34b7e42371a445c1b7f344134bc6e6c0: Deleting consensus metadata
I20260928 05:23:20.457829 13298 raft_consensus.cc:2258] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:20.457911 12790 catalog_manager.cc:4841] TS 34b7e42371a445c1b7f344134bc6e6c0 (127.8.124.195:38117): tablet ad685a5911494641b8e48dfbe26bda46 (table test-table [id=4e567fd6653641ae92f127a0c5d390d0]) successfully deleted
I20260928 05:23:20.458186 13300 ts_tablet_manager.cc:1905] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:20.459735 13298 ts_tablet_manager.cc:1905] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:23:20 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:20.464565 13300 ts_tablet_manager.cc:1918] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:20.464736 13300 log.cc:1196] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/ad685a5911494641b8e48dfbe26bda46
I20260928 05:23:20.465240 13300 ts_tablet_manager.cc:1939] T ad685a5911494641b8e48dfbe26bda46 P 75599bc96dab48c18ff95930268f81ad: Deleting consensus metadata
I20260928 05:23:20.466246 13298 ts_tablet_manager.cc:1918] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:20.466449 13298 log.cc:1196] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/ad685a5911494641b8e48dfbe26bda46
I20260928 05:23:20.466955 13298 ts_tablet_manager.cc:1939] T ad685a5911494641b8e48dfbe26bda46 P 59342cfdba4c4c4c8ffdef7a0ff93ce8: Deleting consensus metadata
I20260928 05:23:20.467021 12790 catalog_manager.cc:4841] TS 75599bc96dab48c18ff95930268f81ad (127.8.124.194:37097): tablet ad685a5911494641b8e48dfbe26bda46 (table test-table [id=4e567fd6653641ae92f127a0c5d390d0]) successfully deleted
I20260928 05:23:20.468253 12789 catalog_manager.cc:4841] TS 59342cfdba4c4c4c8ffdef7a0ff93ce8 (127.8.124.193:40867): tablet ad685a5911494641b8e48dfbe26bda46 (table test-table [id=4e567fd6653641ae92f127a0c5d390d0]) successfully deleted
I20260928 05:23:20.470847 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.433466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.8.124.193:40867)
0928 05:23:20.433696 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.433703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.447668 (+ 13965us) server_negotiation.cc:407] Connection header received
0928 05:23:20.448112 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.448118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.448207 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.448277 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:20.450084 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.451354 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.453902 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.454576 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.454669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.464597 (+ 9928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:20.464611 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:20.464630 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:20.464675 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:20.468163 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.468861 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.468870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.468873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.469014 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.469318 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.469323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.469327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.469824 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:20.470129 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.470308 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.470512 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:23:20.471514 12987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:20.559331 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12865
I20260928 05:23:20.572988 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12998
Sep 28 05:23:20 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:20.587633 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.573879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.8.124.194:37097)
0928 05:23:20.574061 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.574066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.574086 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:23:20.574150 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.574156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.574217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.574354 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:20.575509 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.576258 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.577450 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.577878 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.577987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.582094 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:20.582108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:20.582124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:20.582160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:20.584826 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.585478 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.585486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.585489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.585635 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.586164 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.586170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.586173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.586554 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:20.586811 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.587088 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.587309 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:23:20.588371 13120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:20.673525 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12998
I20260928 05:23:20.686883 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13130
Sep 28 05:23:20 dist-test-slave-515x krb5kdc[12760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:20.702489 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:20.687750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.8.124.195:38117)
0928 05:23:20.687938 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:20.687944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:20.687964 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:23:20.688136 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:20.688142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:20.688209 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:20.688293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:20.689489 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.690355 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.691568 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:20.692028 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:20.692134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:20.696528 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:20.696541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:20.696556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:20.696598 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:20.699601 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.700310 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.700321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.700323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.700484 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:20.700910 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:20.700914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:20.700916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:20.701247 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:20.701556 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:20.701943 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 05:23:20.702140 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:23:20.703224 13252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:20.828410 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13130
I20260928 05:23:20.843357 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 12773
I20260928 05:23:20.844349 12834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:20.948099 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 12773
2026-09-28T05:23:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3733 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T05:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:20Z Disabled control of system clock
I20260928 05:23:21.006242 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40779
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:23:21.311062 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:21.311327 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:21.311448 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:21.318292 13309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:21.318380 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:21.318439 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:21.318509 13309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:21.318563 13309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:21.327461 13309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:40779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:21.328675 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:21.330425 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:21.341064 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:21.341022 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:21.341095 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:21.341956 13309 server_base.cc:1034] running on GCE node
I20260928 05:23:21.342674 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:21.343894 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:21.345093 13309 hybrid_clock.cc:648] HybridClock initialized: now 1790573001345046 us; error 41 us; skew 500 ppm
I20260928 05:23:21.345429 13309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:21.347477 13309 webserver.cc:460] Webserver started at http://127.8.124.254:45527/ using document root <none> and password file <none>
I20260928 05:23:21.348083 13309 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:21.348174 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:21.348498 13309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:21.351136 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "18298979ea404bc09eca353529147563"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "266275b69203ad4be51c5c0c4b9bff0f"
server_key_iv: "8996b5ec61d6130279a56e1efba44ef0"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.351843 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "18298979ea404bc09eca353529147563"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "266275b69203ad4be51c5c0c4b9bff0f"
server_key_iv: "8996b5ec61d6130279a56e1efba44ef0"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.356281 13309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:23:21.359411 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:21.360548 13309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:23:21.360690 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "18298979ea404bc09eca353529147563"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "266275b69203ad4be51c5c0c4b9bff0f"
server_key_iv: "8996b5ec61d6130279a56e1efba44ef0"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.360865 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:21.365942 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:21.366587 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:21.366820 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:21.382563 13309 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:40779
I20260928 05:23:21.382592 13374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:40779 every 8 connection(s)
I20260928 05:23:21.383689 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:21.387097 13375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:21.391449 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13309
I20260928 05:23:21.391621 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:21.392377 8691 external_mini_cluster.cc:1428] Setting key 0c485f9cb8298761cf36762661b1d525
I20260928 05:23:21.397708 13375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: Bootstrap starting.
I20260928 05:23:21.399960 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:21.394245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.8.124.254:40779)
0928 05:23:21.395189 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:21.395200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:21.395240 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:23:21.396810 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:21.396840 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:21.397219 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:21.397529 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:21.398535 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:21.398541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:21.398552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:21.398565 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:21.398622 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:21.398635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:21.398786 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:21.398896 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 05:23:21.399123 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":329,"threads_started":1}
I20260928 05:23:21.400856 13375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:21.402086 13375 log.cc:824] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:21.404503 13375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: No bootstrap required, opened a new log
I20260928 05:23:21.411232 13375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } }
I20260928 05:23:21.411595 13375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:21.411700 13375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18298979ea404bc09eca353529147563, State: Initialized, Role: FOLLOWER
I20260928 05:23:21.412245 13375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } }
I20260928 05:23:21.412446 13375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:21.412568 13375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:21.412720 13375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:21.414722 13375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } }
I20260928 05:23:21.415158 13375 leader_election.cc:302] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18298979ea404bc09eca353529147563; no voters:
I20260928 05:23:21.415623 13375 leader_election.cc:288] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:21.415730 13380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:21.416344 13380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [term 1 LEADER]: Becoming Leader. State: Replica: 18298979ea404bc09eca353529147563, State: Running, Role: LEADER
I20260928 05:23:21.416838 13380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } }
I20260928 05:23:21.417302 13375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:21.420429 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18298979ea404bc09eca353529147563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } } }
I20260928 05:23:21.420451 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18298979ea404bc09eca353529147563. Latest consensus state: current_term: 1 leader_uuid: "18298979ea404bc09eca353529147563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18298979ea404bc09eca353529147563" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40779 } } }
I20260928 05:23:21.420763 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:21.420842 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:21.421525 13389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:21.426760 13389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:21.434715 13389 catalog_manager.cc:1347] Generated new cluster ID: 4169b6a68c02401b9d5adaf127587b81
I20260928 05:23:21.434799 13389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:21.449123 13389 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:21.449486 13389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:21.460762 13389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: Generated new TSK 0
I20260928 05:23:21.461534 13389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:23:21.473583 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--builtin_ntp_servers=127.8.124.212:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:23:21.792752 13399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:21.793002 13399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:21.793114 13399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:21.800096 13399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:21.800184 13399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:21.800323 13399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:21.808890 13399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:21.810055 13399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:21.811735 13399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:21.823458 13404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:21.823501 13405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:21.823589 13407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:21.824402 13399 server_base.cc:1034] running on GCE node
I20260928 05:23:21.824879 13399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:21.825680 13399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:21.826872 13399 hybrid_clock.cc:648] HybridClock initialized: now 1790573001826832 us; error 33 us; skew 500 ppm
I20260928 05:23:21.827152 13399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:21.829094 13399 webserver.cc:460] Webserver started at http://127.8.124.193:41989/ using document root <none> and password file <none>
I20260928 05:23:21.829767 13399 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:21.829855 13399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:21.830137 13399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:21.833758 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "deadcbdcc7694ca28bcab6406abc6014"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "3cb36cca55ec0aaed2ed23792a644322"
server_key_iv: "3ac2832c3bd4aa1cdc99f9ba27df5f56"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.834709 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "deadcbdcc7694ca28bcab6406abc6014"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "3cb36cca55ec0aaed2ed23792a644322"
server_key_iv: "3ac2832c3bd4aa1cdc99f9ba27df5f56"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.840822 13399 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 05:23:21.844945 13413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:21.846419 13399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:21.846563 13399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "deadcbdcc7694ca28bcab6406abc6014"
format_stamp: "Formatted at 2026-09-28 05:23:21 on dist-test-slave-515x"
server_key: "3cb36cca55ec0aaed2ed23792a644322"
server_key_iv: "3ac2832c3bd4aa1cdc99f9ba27df5f56"
server_key_version: "encryptionkey@0"
I20260928 05:23:21.846793 13399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:21.853820 13399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:21.854790 13399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:21.855068 13399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:21.856127 13399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:21.857893 13399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:21.858004 13399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:21.858109 13399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:21.858184 13399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:21.898609 13399 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:36341
I20260928 05:23:21.898761 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:36341 every 8 connection(s)
I20260928 05:23:21.899902 13399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:21.902319 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13399
I20260928 05:23:21.902493 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:21.903311 8691 external_mini_cluster.cc:1428] Setting key 169946e07fc62084f8c70953004e6908
I20260928 05:23:21.905975 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--builtin_ntp_servers=127.8.124.212:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:23:21.908870 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:21.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:53621 (local address 127.8.124.254:40779)
0928 05:23:21.903026 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:21.903030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:21.904569 (+ 1539us) server_negotiation.cc:407] Connection header received
0928 05:23:21.906891 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:21.906896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:21.906964 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:21.907066 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:21.908227 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:21.908232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:21.908236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:21.908252 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:21.908295 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:21.908308 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:21.908421 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:21.908560 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 05:23:21.908613 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":219,"threads_started":1}
I20260928 05:23:21.909776 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:21.903314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40779 (local address 127.8.124.193:53621)
0928 05:23:21.904414 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:21.904465 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:21.906588 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:21.907256 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:21.907264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:21.907358 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:21.907961 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:21.907964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:21.907999 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:21.908081 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:21.908396 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:21.908402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:21.908408 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:23:21.908612 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 05:23:21.908819 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":287,"threads_started":1}
I20260928 05:23:21.911233 13526 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40779
I20260928 05:23:21.911485 13526 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:21.912081 13526 heartbeater.cc:511] Master 127.8.124.254:40779 requested a full tablet report, sending...
I20260928 05:23:21.914004 13340 ts_manager.cc:194] Registered new tserver with Master: deadcbdcc7694ca28bcab6406abc6014 (127.8.124.193:36341)
W20260928 05:23:22.219751 13530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:22.220032 13530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:22.220155 13530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:22.227509 13530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:22.227623 13530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:22.227768 13530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:22.237187 13530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:22.238520 13530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:22.240219 13530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:22.253288 13535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:22.253309 13536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:22.253822 13538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:22.254374 13530 server_base.cc:1034] running on GCE node
I20260928 05:23:22.254952 13530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:22.255848 13530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:22.257035 13530 hybrid_clock.cc:648] HybridClock initialized: now 1790573002256987 us; error 40 us; skew 500 ppm
I20260928 05:23:22.257323 13530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:22.259505 13530 webserver.cc:460] Webserver started at http://127.8.124.194:37865/ using document root <none> and password file <none>
I20260928 05:23:22.260143 13530 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:22.260237 13530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:22.260529 13530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:22.263336 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "cd8879c6d1524ed4af9b7c3480258981"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "23079f8e7e66ffb713ca6238ec95e335"
server_key_iv: "17cfa348fca7269d0c25fd231f9eb6da"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.264144 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "cd8879c6d1524ed4af9b7c3480258981"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "23079f8e7e66ffb713ca6238ec95e335"
server_key_iv: "17cfa348fca7269d0c25fd231f9eb6da"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.268980 13530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:23:22.272207 13544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.273427 13530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:22.273563 13530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "cd8879c6d1524ed4af9b7c3480258981"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "23079f8e7e66ffb713ca6238ec95e335"
server_key_iv: "17cfa348fca7269d0c25fd231f9eb6da"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.273810 13530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:22.278870 13530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:22.279579 13530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:22.279804 13530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:22.280587 13530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:22.281875 13530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:22.281963 13530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.282063 13530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:22.282128 13530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.309966 13530 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:34099
I20260928 05:23:22.310016 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:34099 every 8 connection(s)
I20260928 05:23:22.311161 13530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:22.313930 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13530
I20260928 05:23:22.314079 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:22.314627 8691 external_mini_cluster.cc:1428] Setting key 092db5a4544cd59d39e04812c6bfc91f
I20260928 05:23:22.317236 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--builtin_ntp_servers=127.8.124.212:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:23:22.319033 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.313452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:33491 (local address 127.8.124.254:40779)
0928 05:23:22.313660 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.313665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.315244 (+ 1579us) server_negotiation.cc:407] Connection header received
0928 05:23:22.316938 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.316944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.317002 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.317108 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.318373 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.318379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.318383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.318410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.318447 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.318460 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.318563 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.318711 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.318764 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:23:22.319782 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.314160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40779 (local address 127.8.124.194:33491)
0928 05:23:22.315088 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.315122 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.316688 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.317281 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.317288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.317407 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.318092 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.318094 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.318128 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.318208 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.318538 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.318544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.318549 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.318750 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.318940 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260928 05:23:22.320955 13657 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40779
I20260928 05:23:22.321208 13657 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:22.321768 13657 heartbeater.cc:511] Master 127.8.124.254:40779 requested a full tablet report, sending...
I20260928 05:23:22.323001 13340 ts_manager.cc:194] Registered new tserver with Master: cd8879c6d1524ed4af9b7c3480258981 (127.8.124.194:34099)
W20260928 05:23:22.623581 13660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:22.623840 13660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:22.623957 13660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:22.631222 13660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:22.631337 13660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:22.631491 13660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:22.640429 13660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:22.641590 13660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:22.643271 13660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:22.655486 13665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:22.655524 13668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:22.655548 13666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:22.656368 13660 server_base.cc:1034] running on GCE node
I20260928 05:23:22.656942 13660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:22.657837 13660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:22.659037 13660 hybrid_clock.cc:648] HybridClock initialized: now 1790573002658986 us; error 40 us; skew 500 ppm
I20260928 05:23:22.659327 13660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:22.661446 13660 webserver.cc:460] Webserver started at http://127.8.124.195:39191/ using document root <none> and password file <none>
I20260928 05:23:22.662135 13660 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:22.662230 13660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:22.662514 13660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:22.665266 13660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "e82d299ae79246a6ad1a2a715a4c9e21"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "21d4b2eaec0ed0e1443023031af6ae63"
server_key_iv: "897f57fc92e8929b4035211e11577206"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.666007 13660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "e82d299ae79246a6ad1a2a715a4c9e21"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "21d4b2eaec0ed0e1443023031af6ae63"
server_key_iv: "897f57fc92e8929b4035211e11577206"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.670640 13660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 05:23:22.673840 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.675100 13660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:22.675237 13660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "e82d299ae79246a6ad1a2a715a4c9e21"
format_stamp: "Formatted at 2026-09-28 05:23:22 on dist-test-slave-515x"
server_key: "21d4b2eaec0ed0e1443023031af6ae63"
server_key_iv: "897f57fc92e8929b4035211e11577206"
server_key_version: "encryptionkey@0"
I20260928 05:23:22.675431 13660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:22.680577 13660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:22.681265 13660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:22.681485 13660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:22.682348 13660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:22.683574 13660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:22.683662 13660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.683759 13660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:22.683830 13660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:22.712113 13660 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:36283
I20260928 05:23:22.712169 13786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:36283 every 8 connection(s)
I20260928 05:23:22.713364 13660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:22.713726 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13660
I20260928 05:23:22.713892 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:22.714437 8691 external_mini_cluster.cc:1428] Setting key 0bfe98c0c624facb6e1a092930dc8449
I20260928 05:23:22.721467 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.715979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:39985 (local address 127.8.124.254:40779)
0928 05:23:22.716133 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.716138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.717884 (+ 1746us) server_negotiation.cc:407] Connection header received
0928 05:23:22.719526 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.719531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.719590 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.719669 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.720816 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.720821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.720827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.720843 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.720871 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.720881 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.720977 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.721154 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.721211 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:22.722330 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.716685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:40779 (local address 127.8.124.195:39985)
0928 05:23:22.717732 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.717790 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.719299 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.719880 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.719887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.719983 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.720555 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.720557 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.720593 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.720667 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.720974 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.720983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.720990 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.721218 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.721441 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":296,"threads_started":1}
I20260928 05:23:22.723500 13787 heartbeater.cc:347] Connected to a master server at 127.8.124.254:40779
I20260928 05:23:22.723721 13787 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:22.724196 13787 heartbeater.cc:511] Master 127.8.124.254:40779 requested a full tablet report, sending...
I20260928 05:23:22.725284 13340 ts_manager.cc:194] Registered new tserver with Master: e82d299ae79246a6ad1a2a715a4c9e21 (127.8.124.195:36283)
I20260928 05:23:22.729161 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:23:22.735157 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.733162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.8.124.254:40779)
0928 05:23:22.733280 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.733284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.733741 (+ 457us) server_negotiation.cc:407] Connection header received
0928 05:23:22.733858 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.733861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.733904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.733973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.734701 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.734704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.734708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.734720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.734746 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.734755 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.734839 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.734886 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.734925 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 05:23:22.738528 13340 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:22.744757 13340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:22.761247 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.755545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:34099 (local address 127.0.0.1:43792)
0928 05:23:22.756333 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.756374 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.756493 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.758416 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.758428 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.758538 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.759283 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.759287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.759319 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.759391 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.760579 (+ 1188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.760588 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.760595 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.760774 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.760857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":1333,"threads_started":1}
I20260928 05:23:22.761392 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.755746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.8.124.194:34099)
0928 05:23:22.757883 (+ 2137us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.757891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.757922 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:23:22.757986 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.757994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.758218 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.759227 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.760380 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.760388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.760401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.760422 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.760460 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.760477 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.760632 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.760995 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.761081 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1833,"threads_started":1}
I20260928 05:23:22.761760 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.756502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.8.124.193:36341)
0928 05:23:22.756989 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.756998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.758823 (+ 1825us) server_negotiation.cc:407] Connection header received
0928 05:23:22.758977 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.758983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.759173 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.759284 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.760989 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.760997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.761008 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.761025 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.761078 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.761102 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.761197 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.761398 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.761487 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":198,"threads_started":1}
I20260928 05:23:22.762105 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.756187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.8.124.195:36283)
0928 05:23:22.757405 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.757413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.758651 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 05:23:22.758717 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.758722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.758908 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.759044 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.760391 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.760399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.760411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.760429 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.760483 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.760502 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.760646 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.761683 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.761770 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":920,"threads_started":1}
I20260928 05:23:22.762408 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.756036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59446)
0928 05:23:22.758713 (+ 2677us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.758728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.758829 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.759785 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.759789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.759809 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.760763 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.760766 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.760790 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.760831 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.761268 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.761271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.761274 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.762019 (+ 745us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.762060 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"thread_start_us":2129,"threads_started":1}
I20260928 05:23:22.762612 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.755663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36283 (local address 127.0.0.1:32904)
0928 05:23:22.758437 (+ 2774us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.758458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.758563 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.759055 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.759060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.759081 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.760168 (+ 1087us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.760171 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.760186 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.760224 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.760693 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.760698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.760702 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.760823 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.760865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2628,"thread_start_us":2806,"threads_started":1}
I20260928 05:23:22.764909 13592 tablet_service.cc:1467] Processing CreateTablet for tablet cb1917d2eb87416598e0cec2a30418e7 (DEFAULT_TABLE table=test-table [id=3349535b4348453396e3edadab836655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:22.765686 13722 tablet_service.cc:1467] Processing CreateTablet for tablet cb1917d2eb87416598e0cec2a30418e7 (DEFAULT_TABLE table=test-table [id=3349535b4348453396e3edadab836655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:22.765686 13461 tablet_service.cc:1467] Processing CreateTablet for tablet cb1917d2eb87416598e0cec2a30418e7 (DEFAULT_TABLE table=test-table [id=3349535b4348453396e3edadab836655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:22.766250 13592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1917d2eb87416598e0cec2a30418e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:22.766923 13461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1917d2eb87416598e0cec2a30418e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:22.766924 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1917d2eb87416598e0cec2a30418e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:22.776633 13805 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Bootstrap starting.
I20260928 05:23:22.779731 13805 tablet_bootstrap.cc:654] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:22.780829 13805 log.cc:824] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:22.783259 13805 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: No bootstrap required, opened a new log
I20260928 05:23:22.783473 13805 ts_tablet_manager.cc:1397] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 05:23:22.785744 13807 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Bootstrap starting.
I20260928 05:23:22.786322 13806 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Bootstrap starting.
I20260928 05:23:22.789132 13806 tablet_bootstrap.cc:654] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:22.789945 13807 tablet_bootstrap.cc:654] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:22.790200 13806 log.cc:824] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:22.791061 13807 log.cc:824] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:22.792045 13805 raft_consensus.cc:348] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.792402 13805 raft_consensus.cc:374] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:22.792496 13805 raft_consensus.cc:729] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8879c6d1524ed4af9b7c3480258981, State: Initialized, Role: FOLLOWER
I20260928 05:23:22.793004 13806 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: No bootstrap required, opened a new log
I20260928 05:23:22.793247 13806 ts_tablet_manager.cc:1397] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:23:22.793108 13805 consensus_queue.cc:260] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.793924 13807 tablet_bootstrap.cc:492] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: No bootstrap required, opened a new log
I20260928 05:23:22.794152 13657 heartbeater.cc:503] Master 127.8.124.254:40779 was elected leader, sending a full tablet report...
I20260928 05:23:22.794196 13807 ts_tablet_manager.cc:1397] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20260928 05:23:22.794731 13805 ts_tablet_manager.cc:1428] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260928 05:23:22.801088 13806 raft_consensus.cc:348] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.801394 13806 raft_consensus.cc:374] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:22.801486 13806 raft_consensus.cc:729] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82d299ae79246a6ad1a2a715a4c9e21, State: Initialized, Role: FOLLOWER
I20260928 05:23:22.802026 13806 consensus_queue.cc:260] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.802218 13807 raft_consensus.cc:348] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.802610 13807 raft_consensus.cc:374] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:22.802706 13807 raft_consensus.cc:729] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadcbdcc7694ca28bcab6406abc6014, State: Initialized, Role: FOLLOWER
I20260928 05:23:22.803009 13787 heartbeater.cc:503] Master 127.8.124.254:40779 was elected leader, sending a full tablet report...
I20260928 05:23:22.803226 13807 consensus_queue.cc:260] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.803557 13806 ts_tablet_manager.cc:1428] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260928 05:23:22.804277 13526 heartbeater.cc:503] Master 127.8.124.254:40779 was elected leader, sending a full tablet report...
I20260928 05:23:22.804637 13807 ts_tablet_manager.cc:1428] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
W20260928 05:23:22.812745 13658 tablet.cc:2369] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:22.893765 13813 raft_consensus.cc:482] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:22.894026 13813 raft_consensus.cc:504] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.895545 13813 leader_election.cc:288] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd8879c6d1524ed4af9b7c3480258981 (127.8.124.194:34099), e82d299ae79246a6ad1a2a715a4c9e21 (127.8.124.195:36283)
I20260928 05:23:22.899005 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.895885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:37831 (local address 127.8.124.195:36283)
0928 05:23:22.896043 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.896050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.896933 (+ 883us) server_negotiation.cc:407] Connection header received
0928 05:23:22.897119 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.897124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.897182 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.897276 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.898406 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.898411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.898415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.898434 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.898465 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.898477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.898581 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.898692 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.898750 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:23:22.899078 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.895928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:33275 (local address 127.8.124.194:34099)
0928 05:23:22.896083 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.896089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.896972 (+ 883us) server_negotiation.cc:407] Connection header received
0928 05:23:22.897118 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.897123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.897182 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.897279 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.898459 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.898465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.898470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.898490 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.898524 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.898538 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.898645 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.898740 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.898795 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:22.899309 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.896208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:34099 (local address 127.8.124.193:33275)
0928 05:23:22.896885 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.896898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.897002 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.897319 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.897323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.897372 (+ 49us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.898211 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.898215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.898234 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.898271 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.898637 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.898642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.898646 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.898777 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.898824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"spinlock_wait_cycles":7808,"thread_start_us":394,"threads_started":1}
I20260928 05:23:22.899726 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.895943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36283 (local address 127.8.124.193:37831)
0928 05:23:22.896814 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:22.896840 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:22.897002 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:22.897316 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:22.897321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:22.897345 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:22.898161 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:23:22.898164 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:22.898181 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:23:22.898231 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:22.898557 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:22.898561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:22.898566 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:23:22.899226 (+ 660us) client_negotiation.cc:240] Negotiation successful
0928 05:23:22.899277 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"spinlock_wait_cycles":46080,"thread_start_us":424,"threads_started":1}
I20260928 05:23:22.900449 13612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1917d2eb87416598e0cec2a30418e7" candidate_uuid: "deadcbdcc7694ca28bcab6406abc6014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8879c6d1524ed4af9b7c3480258981" is_pre_election: true
I20260928 05:23:22.900671 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1917d2eb87416598e0cec2a30418e7" candidate_uuid: "deadcbdcc7694ca28bcab6406abc6014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" is_pre_election: true
I20260928 05:23:22.900875 13612 raft_consensus.cc:2454] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadcbdcc7694ca28bcab6406abc6014 in term 0.
I20260928 05:23:22.901170 13742 raft_consensus.cc:2454] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadcbdcc7694ca28bcab6406abc6014 in term 0.
I20260928 05:23:22.901435 13414 leader_election.cc:302] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd8879c6d1524ed4af9b7c3480258981, deadcbdcc7694ca28bcab6406abc6014; no voters:
I20260928 05:23:22.901813 13813 raft_consensus.cc:2781] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260928 05:23:22.901965 13527 tablet.cc:2369] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:22.901990 13813 raft_consensus.cc:482] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:22.902191 13813 raft_consensus.cc:3037] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:22.904305 13813 raft_consensus.cc:504] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.904879 13813 leader_election.cc:288] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [CANDIDATE]: Term 1 election: Requested vote from peers cd8879c6d1524ed4af9b7c3480258981 (127.8.124.194:34099), e82d299ae79246a6ad1a2a715a4c9e21 (127.8.124.195:36283)
I20260928 05:23:22.905201 13612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1917d2eb87416598e0cec2a30418e7" candidate_uuid: "deadcbdcc7694ca28bcab6406abc6014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8879c6d1524ed4af9b7c3480258981"
I20260928 05:23:22.905339 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1917d2eb87416598e0cec2a30418e7" candidate_uuid: "deadcbdcc7694ca28bcab6406abc6014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82d299ae79246a6ad1a2a715a4c9e21"
I20260928 05:23:22.905395 13612 raft_consensus.cc:3037] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:22.905524 13742 raft_consensus.cc:3037] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:22.908377 13612 raft_consensus.cc:2454] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadcbdcc7694ca28bcab6406abc6014 in term 1.
I20260928 05:23:22.908429 13742 raft_consensus.cc:2454] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadcbdcc7694ca28bcab6406abc6014 in term 1.
I20260928 05:23:22.908722 13414 leader_election.cc:302] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd8879c6d1524ed4af9b7c3480258981, deadcbdcc7694ca28bcab6406abc6014; no voters:
I20260928 05:23:22.908984 13813 raft_consensus.cc:2781] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:22.909682 13813 raft_consensus.cc:686] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 1 LEADER]: Becoming Leader. State: Replica: deadcbdcc7694ca28bcab6406abc6014, State: Running, Role: LEADER
I20260928 05:23:22.910377 13813 consensus_queue.cc:237] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } }
I20260928 05:23:22.914640 13339 catalog_manager.cc:5495] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 reported cstate change: term changed from 0 to 1, leader changed from <none> to deadcbdcc7694ca28bcab6406abc6014 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "deadcbdcc7694ca28bcab6406abc6014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deadcbdcc7694ca28bcab6406abc6014" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36341 } health_report { overall_health: HEALTHY } } }
I20260928 05:23:22.954586 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.952723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.8.124.193:36341)
0928 05:23:22.952874 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.952878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:22.952894 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:22.953023 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:22.953026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:22.953112 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:22.953192 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:22.954044 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:22.954048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:22.954051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:22.954067 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:22.954098 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:22.954110 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:22.954207 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:22.954265 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 05:23:22.954308 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:23:22.962087 13742 raft_consensus.cc:1261] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 1 FOLLOWER]: Refusing update from remote peer deadcbdcc7694ca28bcab6406abc6014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:22.962087 13612 raft_consensus.cc:1261] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 1 FOLLOWER]: Refusing update from remote peer deadcbdcc7694ca28bcab6406abc6014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:22.962965 13816 consensus_queue.cc:1035] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8879c6d1524ed4af9b7c3480258981" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:22.963304 13813 consensus_queue.cc:1035] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82d299ae79246a6ad1a2a715a4c9e21" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:23:22.967029 13788 tablet.cc:2369] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:22.972512 13819 mvcc.cc:205] Tried to move back new op lower bound from 7334187020123840512 to 7334187019924336640. Current Snapshot: MvccSnapshot[applied={T|T < 7334187020123840512}]
I20260928 05:23:22.974225 13820 mvcc.cc:205] Tried to move back new op lower bound from 7334187020123840512 to 7334187019924336640. Current Snapshot: MvccSnapshot[applied={T|T < 7334187020123840512}]
I20260928 05:23:22.986232 13339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:22.986483 13339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:22.990020 13339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18298979ea404bc09eca353529147563: Sending DeleteTablet for 3 replicas of tablet cb1917d2eb87416598e0cec2a30418e7
I20260928 05:23:22.991135 13592 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1917d2eb87416598e0cec2a30418e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:22 UTC) from {username='slave'} at 127.0.0.1:43792
I20260928 05:23:22.991315 13722 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1917d2eb87416598e0cec2a30418e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:22 UTC) from {username='slave'} at 127.0.0.1:32904
I20260928 05:23:22.991853 13461 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1917d2eb87416598e0cec2a30418e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:22 UTC) from {username='slave'} at 127.0.0.1:59446
I20260928 05:23:22.993189 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13399
I20260928 05:23:23.001720 13829 tablet_replica.cc:330] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: stopping tablet replica
I20260928 05:23:23.002185 13829 raft_consensus.cc:2229] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:23.002699 13829 raft_consensus.cc:2258] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:23.004390 13831 tablet_replica.cc:330] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014: stopping tablet replica
I20260928 05:23:23.004794 13831 raft_consensus.cc:2229] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:23.005425 13831 raft_consensus.cc:2258] T cb1917d2eb87416598e0cec2a30418e7 P deadcbdcc7694ca28bcab6406abc6014 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:23.008217 13829 ts_tablet_manager.cc:1905] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:23.009055 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:22.996076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.8.124.193:36341)
0928 05:23:22.996232 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:22.996238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:23.006203 (+ 9965us) server_negotiation.cc:407] Connection header received
0928 05:23:23.006558 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:23.006564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:23.006629 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:23.006701 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:23.008237 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:23.008243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:23.008249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:23.008269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:23.008309 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:23.008325 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:23.008427 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:23.008651 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 05:23:23.008719 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:23:23.010120 13521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:23.011170 13830 tablet_replica.cc:330] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: stopping tablet replica
I20260928 05:23:23.011549 13830 raft_consensus.cc:2229] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:23.012004 13830 raft_consensus.cc:2258] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:23.014266 13830 ts_tablet_manager.cc:1905] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:23.016152 13829 ts_tablet_manager.cc:1918] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:23.016299 13829 log.cc:1196] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/cb1917d2eb87416598e0cec2a30418e7
I20260928 05:23:23.016742 13829 ts_tablet_manager.cc:1939] T cb1917d2eb87416598e0cec2a30418e7 P cd8879c6d1524ed4af9b7c3480258981: Deleting consensus metadata
I20260928 05:23:23.018440 13324 catalog_manager.cc:4841] TS cd8879c6d1524ed4af9b7c3480258981 (127.8.124.194:34099): tablet cb1917d2eb87416598e0cec2a30418e7 (table test-table [id=3349535b4348453396e3edadab836655]) successfully deleted
I20260928 05:23:23.019461 13830 ts_tablet_manager.cc:1918] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:23.019585 13830 log.cc:1196] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/cb1917d2eb87416598e0cec2a30418e7
I20260928 05:23:23.019970 13830 ts_tablet_manager.cc:1939] T cb1917d2eb87416598e0cec2a30418e7 P e82d299ae79246a6ad1a2a715a4c9e21: Deleting consensus metadata
I20260928 05:23:23.021227 13327 catalog_manager.cc:4841] TS e82d299ae79246a6ad1a2a715a4c9e21 (127.8.124.195:36283): tablet cb1917d2eb87416598e0cec2a30418e7 (table test-table [id=3349535b4348453396e3edadab836655]) successfully deleted
I20260928 05:23:23.104142 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13399
W20260928 05:23:23.118170 13325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:36341 (error 108)
I20260928 05:23:23.118398 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13530
I20260928 05:23:23.120015 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.119576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59476)
0928 05:23:23.119727 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.119855 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:23:23.120256 13325 catalog_manager.cc:4584] TS deadcbdcc7694ca28bcab6406abc6014 (127.8.124.193:36341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb1917d2eb87416598e0cec2a30418e7: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
I20260928 05:23:23.121148 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.119315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.8.124.194:34099)
0928 05:23:23.119484 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:23.119490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:23.119508 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:23.119571 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:23.119577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:23.119682 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:23.119777 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:23.120698 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:23.120703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:23.120706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:23.120722 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:23.120753 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:23.120765 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:23.120876 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:23.120911 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 05:23:23.120953 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:23:23.121819 13652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:23.174134 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.173680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59484)
0928 05:23:23.173856 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.173968 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:23:23.175099 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.174774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59488)
0928 05:23:23.174921 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.174990 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:23:23.208747 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13530
I20260928 05:23:23.223444 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13660
I20260928 05:23:23.225888 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.224119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.8.124.195:36283)
0928 05:23:23.224275 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:23.224279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:23.224296 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:23.224447 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:23.224451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:23.224523 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:23.224603 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:23.225376 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:23.225380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:23.225383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:23.225398 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:23.225430 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:23.225441 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:23.225558 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:23.225592 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 05:23:23.225675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:23:23.226462 13782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:23.235299 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.234790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59502)
0928 05:23:23.235003 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.235127 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:23:23.236265 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.235965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59506)
0928 05:23:23.236092 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.236158 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:23:23.311453 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13660
I20260928 05:23:23.316701 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.316207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59512)
0928 05:23:23.316423 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.316537 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:23:23.317796 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.317316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36341 (local address 127.0.0.1:59514)
0928 05:23:23.317528 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.317644 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 05:23:23.324591 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13309
I20260928 05:23:23.325644 13370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:23.426940 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13309
2026-09-28T05:23:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2478 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 05:23:23.462695 8691 mini_oidc.cc:152] Starting JWKS server
I20260928 05:23:23.465135 8691 webserver.cc:460] Webserver started at https://127.0.0.1:43425/ using document root <none> and password file <none>
I20260928 05:23:23.465446 8691 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:23:23.466111 8691 webserver.cc:460] Webserver started at http://127.0.0.1:45651/ using document root <none> and password file <none>
2026-09-28T05:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:23Z Disabled control of system clock
I20260928 05:23:23.489841 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:39961
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:39961
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/testchainca.cert with env {}
W20260928 05:23:23.790813 13843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:23.791096 13843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:23.791178 13843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:23.798404 13843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:23.798512 13843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:23.798547 13843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:23.798601 13843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:23.798643 13843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:23:23.798702 13843 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:23:23.798738 13843 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43425/jwks.json
I20260928 05:23:23.808748 13843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:39961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:23.810024 13843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:23.811266 13843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:23.822335 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:23.822372 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:23.822424 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:23.822942 13843 server_base.cc:1034] running on GCE node
I20260928 05:23:23.823668 13843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:23.825026 13843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:23.826215 13843 hybrid_clock.cc:648] HybridClock initialized: now 1790573003826175 us; error 33 us; skew 500 ppm
I20260928 05:23:23.826551 13843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:23.833746 13843 webserver.cc:460] Webserver started at http://127.8.124.254:35239/ using document root <none> and password file <none>
I20260928 05:23:23.834470 13843 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:23.834575 13843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:23.834873 13843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:23.837738 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "5d6b0b865947489694f2e63d8b46144a"
format_stamp: "Formatted at 2026-09-28 05:23:23 on dist-test-slave-515x"
server_key: "dd3625ef0b1740210a705809110506b5"
server_key_iv: "05becdd05aa5f90ccb6b6ff171d034c1"
server_key_version: "encryptionkey@0"
I20260928 05:23:23.838505 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "5d6b0b865947489694f2e63d8b46144a"
format_stamp: "Formatted at 2026-09-28 05:23:23 on dist-test-slave-515x"
server_key: "dd3625ef0b1740210a705809110506b5"
server_key_iv: "05becdd05aa5f90ccb6b6ff171d034c1"
server_key_version: "encryptionkey@0"
I20260928 05:23:23.843390 13843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:23:23.846642 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:23.848078 13843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:23.848229 13843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "5d6b0b865947489694f2e63d8b46144a"
format_stamp: "Formatted at 2026-09-28 05:23:23 on dist-test-slave-515x"
server_key: "dd3625ef0b1740210a705809110506b5"
server_key_iv: "05becdd05aa5f90ccb6b6ff171d034c1"
server_key_version: "encryptionkey@0"
I20260928 05:23:23.848414 13843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:23.881150 13843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:23.882012 13843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:23.882290 13843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:23.898566 13843 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:39961
I20260928 05:23:23.898587 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:39961 every 8 connection(s)
I20260928 05:23:23.899809 13843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:23.903375 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:23.905728 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13843
I20260928 05:23:23.905942 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:23.906683 8691 external_mini_cluster.cc:1428] Setting key f71c0fc5213d6a0b205a72233b2f2c9f
I20260928 05:23:23.914795 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a: Bootstrap starting.
I20260928 05:23:23.917673 13909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:23.918707 13909 log.cc:824] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:23.919245 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.908576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.8.124.254:39961)
0928 05:23:23.909486 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:23.909497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:23.909538 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:23:23.911156 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:23.911177 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:23.911618 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:23.911972 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:23.912533 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:23.913828 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:23.915123 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:23.915750 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:23.915917 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:23.917021 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:23.917028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:23.917041 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:23.917531 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:23.917668 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:23.917690 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:23.917789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:23.918001 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 05:23:23.918276 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260928 05:23:23.921445 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a: No bootstrap required, opened a new log
I20260928 05:23:23.922663 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:23:23.928411 13909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } }
I20260928 05:23:23.928725 13909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:23.928829 13909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6b0b865947489694f2e63d8b46144a, State: Initialized, Role: FOLLOWER
I20260928 05:23:23.929371 13909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } }
I20260928 05:23:23.929574 13909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:23.929713 13909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:23.929854 13909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:23.931843 13909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } }
I20260928 05:23:23.932296 13909 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6b0b865947489694f2e63d8b46144a; no voters:
I20260928 05:23:23.932843 13909 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:23.932936 13914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:23.933501 13914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [term 1 LEADER]: Becoming Leader. State: Replica: 5d6b0b865947489694f2e63d8b46144a, State: Running, Role: LEADER
I20260928 05:23:23.934001 13914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } }
I20260928 05:23:23.934955 13909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:23.936856 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6b0b865947489694f2e63d8b46144a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } } }
I20260928 05:23:23.937242 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:23.937301 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6b0b865947489694f2e63d8b46144a. Latest consensus state: current_term: 1 leader_uuid: "5d6b0b865947489694f2e63d8b46144a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b0b865947489694f2e63d8b46144a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39961 } } }
I20260928 05:23:23.937588 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:23.938189 13920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:23.942945 13920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:23.951524 13920 catalog_manager.cc:1347] Generated new cluster ID: d742a8d5de474def937e8ca46f1d3258
I20260928 05:23:23.951637 13920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:23.960633 13920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:23.962013 13920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:23.974153 13920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6b0b865947489694f2e63d8b46144a: Generated new TSK 0
I20260928 05:23:23.975256 13920 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:23:24.026127 13939 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 05:23:24.026292 13939 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:23:24.027000 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:23.998963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.8.124.254:39961)
0928 05:23:23.999140 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:23.999148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:23.999520 (+ 372us) server_negotiation.cc:407] Connection header received
0928 05:23:23.999814 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:23.999818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:23.999911 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:24.000029 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:23:24.000591 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.001317 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.003396 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.003832 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.003932 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:24.004518 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:23:24.026501 (+ 21983us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:23:24.026773 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 05:23:24.027103 13912 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 05:23:24.028847 13938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:23.998755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:39961 (local address 127.0.0.1:42256)
0928 05:23:23.999411 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:23.999434 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:23.999612 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:24.000077 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:24.000084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:24.000097 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:23:24.000457 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:24.000470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.001457 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.001464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:24.003246 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:24.003261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.004000 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.004007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:24.004193 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:24.004268 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:23:24.026774 (+ 22506us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:23:24.027123 (+ 349us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.124.254:39961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":300,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.124.254:39961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 05:23:24.030567 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13843
I20260928 05:23:24.031761 13904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:24.121658 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13843
2026-09-28T05:23:24Z chronyd exiting
I20260928 05:23:24.168552 8691 test_util.cc:175] -----------------------------------------------
I20260928 05:23:24.168680 8691 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572965871275-8691-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 05:23:24.172232 8691 mini_oidc.cc:152] Starting JWKS server
I20260928 05:23:24.173952 8691 webserver.cc:460] Webserver started at https://127.0.0.1:34671/ using document root <none> and password file <none>
I20260928 05:23:24.174141 8691 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:23:24.174710 8691 webserver.cc:460] Webserver started at http://127.0.0.1:36661/ using document root <none> and password file <none>
2026-09-28T05:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:24Z Disabled control of system clock
I20260928 05:23:24.197800 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45463
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/testchainca.cert with env {}
W20260928 05:23:24.499826 13946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:24.500098 13946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:24.500221 13946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:24.507308 13946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:24.507416 13946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:24.507473 13946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:24.507550 13946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:24.507602 13946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:23:24.507682 13946 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:23:24.507725 13946 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34671/jwks.json
I20260928 05:23:24.517401 13946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:24.518669 13946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:24.519934 13946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:24.530544 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:24.530593 13952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:24.530620 13954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:24.531440 13946 server_base.cc:1034] running on GCE node
I20260928 05:23:24.532191 13946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:24.533499 13946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:24.534683 13946 hybrid_clock.cc:648] HybridClock initialized: now 1790573004534640 us; error 36 us; skew 500 ppm
I20260928 05:23:24.534989 13946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:24.537041 13946 webserver.cc:460] Webserver started at http://127.8.124.254:43867/ using document root <none> and password file <none>
I20260928 05:23:24.537703 13946 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:24.537796 13946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:24.538084 13946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:24.540699 13946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "4554ffabe1914710bae104647c00c6ba"
format_stamp: "Formatted at 2026-09-28 05:23:24 on dist-test-slave-515x"
server_key: "7632876015540e2f1dbdfd49085226b8"
server_key_iv: "0b5ccf625cca103b571385065418dec9"
server_key_version: "encryptionkey@0"
I20260928 05:23:24.541404 13946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "4554ffabe1914710bae104647c00c6ba"
format_stamp: "Formatted at 2026-09-28 05:23:24 on dist-test-slave-515x"
server_key: "7632876015540e2f1dbdfd49085226b8"
server_key_iv: "0b5ccf625cca103b571385065418dec9"
server_key_version: "encryptionkey@0"
I20260928 05:23:24.546042 13946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:23:24.549094 13960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:24.550406 13946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:24.550549 13946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "4554ffabe1914710bae104647c00c6ba"
format_stamp: "Formatted at 2026-09-28 05:23:24 on dist-test-slave-515x"
server_key: "7632876015540e2f1dbdfd49085226b8"
server_key_iv: "0b5ccf625cca103b571385065418dec9"
server_key_version: "encryptionkey@0"
I20260928 05:23:24.550724 13946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:24.566735 13946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:24.567451 13946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:24.567705 13946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:24.584240 13946 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45463
I20260928 05:23:24.584264 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45463 every 8 connection(s)
I20260928 05:23:24.585475 13946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:24.588941 14012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:24.593015 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 13946
I20260928 05:23:24.593209 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:24.594044 8691 external_mini_cluster.cc:1428] Setting key 5c18ad4a3f7e24053797d76322780c92
I20260928 05:23:24.599924 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba: Bootstrap starting.
I20260928 05:23:24.602923 14012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:24.603884 14012 log.cc:824] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:24.606992 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba: No bootstrap required, opened a new log
I20260928 05:23:24.607118 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:24.595975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.8.124.254:45463)
0928 05:23:24.596907 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:24.596918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:24.596957 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:23:24.598744 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:24.598765 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:24.599201 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:24.599531 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:24.600053 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.601327 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.602574 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.603268 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.603434 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:24.604612 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:24.604619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:24.604631 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:24.605190 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:24.605282 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:24.605327 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:24.605435 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:24.605647 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 05:23:24.605948 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":316,"threads_started":1}
I20260928 05:23:24.610555 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:23:24.615185 14012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } }
I20260928 05:23:24.615489 14012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:24.615559 14012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4554ffabe1914710bae104647c00c6ba, State: Initialized, Role: FOLLOWER
I20260928 05:23:24.616055 14012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } }
I20260928 05:23:24.616246 14012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:24.616338 14012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:24.616448 14012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:24.618458 14012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } }
I20260928 05:23:24.618871 14012 leader_election.cc:302] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4554ffabe1914710bae104647c00c6ba; no voters:
I20260928 05:23:24.619333 14012 leader_election.cc:288] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:24.619467 14017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:24.620484 14017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [term 1 LEADER]: Becoming Leader. State: Replica: 4554ffabe1914710bae104647c00c6ba, State: Running, Role: LEADER
I20260928 05:23:24.620958 14017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } }
I20260928 05:23:24.621217 14012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:24.623523 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4554ffabe1914710bae104647c00c6ba. Latest consensus state: current_term: 1 leader_uuid: "4554ffabe1914710bae104647c00c6ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } } }
I20260928 05:23:24.623890 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:24.623730 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4554ffabe1914710bae104647c00c6ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ffabe1914710bae104647c00c6ba" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45463 } } }
I20260928 05:23:24.624040 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:24.625391 14026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:24.630028 14026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:24.638569 14026 catalog_manager.cc:1347] Generated new cluster ID: a835ab1a1e5542ad8be134f2ad275ee8
I20260928 05:23:24.638672 14026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:24.659189 14026 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:24.660409 14026 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:24.665464 14026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4554ffabe1914710bae104647c00c6ba: Generated new TSK 0
I20260928 05:23:24.666251 14026 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:23:24.694442 14042 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 05:23:24.694550 14042 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:23:24.695745 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:24.676404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.8.124.254:45463)
0928 05:23:24.676581 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:24.676588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:24.676874 (+ 286us) server_negotiation.cc:407] Connection header received
0928 05:23:24.677079 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:24.677084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:24.677177 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:24.677335 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:23:24.677796 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.678880 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.680191 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.680857 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.680991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:24.681380 (+ 389us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:23:24.695088 (+ 13708us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:23:24.695463 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 05:23:24.695793 14041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:24.676279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45463 (local address 127.0.0.1:41712)
0928 05:23:24.676777 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:24.676793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:24.676903 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:24.677369 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:24.677373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:24.677382 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:23:24.677646 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:24.677657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.679004 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.679009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:24.680063 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:24.680074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:24.681006 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:24.681010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:24.681136 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:24.681166 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:23:24.695317 (+ 14151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:23:24.695514 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.124.254:45463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":269,"threads_started":1}
W20260928 05:23:24.695864 14015 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 05:23:24.696987 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 13946
I20260928 05:23:24.698307 14007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:24.786597 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 13946
2026-09-28T05:23:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (665 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 05:23:24.837726 8691 mini_oidc.cc:152] Starting JWKS server
I20260928 05:23:24.839483 8691 webserver.cc:460] Webserver started at https://127.0.0.1:36229/ using document root <none> and password file <none>
I20260928 05:23:24.839704 8691 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:23:24.840279 8691 webserver.cc:460] Webserver started at http://127.0.0.1:44483/ using document root <none> and password file <none>
2026-09-28T05:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:24Z Disabled control of system clock
I20260928 05:23:24.863495 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46823
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46823
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:23:25.162117 14049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:25.162386 14049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:25.162500 14049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:25.169404 14049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:25.169507 14049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:25.169548 14049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:25.169638 14049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:25.169699 14049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:23:25.169783 14049 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:23:25.169834 14049 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36229/jwks.json
I20260928 05:23:25.178825 14049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:25.180037 14049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:25.182210 14049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:25.193817 14057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:25.193926 14054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:25.193873 14055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:25.195031 14049 server_base.cc:1034] running on GCE node
I20260928 05:23:25.195808 14049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:25.197140 14049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:25.198338 14049 hybrid_clock.cc:648] HybridClock initialized: now 1790573005198287 us; error 55 us; skew 500 ppm
I20260928 05:23:25.198686 14049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:25.200762 14049 webserver.cc:460] Webserver started at http://127.8.124.254:45017/ using document root <none> and password file <none>
I20260928 05:23:25.201411 14049 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:25.201493 14049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:25.201821 14049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:25.204569 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "e48e958b143e4e5a86b26256b1e9e33e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "4274df711f75b7a3e5d54137966559ba"
server_key_iv: "6bcc4459c47181b6809b009e9efba3f8"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.205317 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "e48e958b143e4e5a86b26256b1e9e33e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "4274df711f75b7a3e5d54137966559ba"
server_key_iv: "6bcc4459c47181b6809b009e9efba3f8"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.210176 14049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:23:25.213506 14063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:25.214866 14049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:23:25.215022 14049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "e48e958b143e4e5a86b26256b1e9e33e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "4274df711f75b7a3e5d54137966559ba"
server_key_iv: "6bcc4459c47181b6809b009e9efba3f8"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.215206 14049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:25.235702 14049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:25.236548 14049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:25.236814 14049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:25.253176 14049 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:46823
I20260928 05:23:25.253186 14114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:46823 every 8 connection(s)
I20260928 05:23:25.254446 14049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:25.257723 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14049
I20260928 05:23:25.257918 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:25.258097 14115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:25.258672 8691 external_mini_cluster.cc:1428] Setting key 685ef55b355f9d89cfff6b1dbc4f7390
I20260928 05:23:25.272039 14115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e: Bootstrap starting.
I20260928 05:23:25.274107 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:25.261744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.8.124.254:46823)
0928 05:23:25.262824 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:25.262838 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:25.262884 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:23:25.264734 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:25.264761 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:25.265370 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:25.265815 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:25.266310 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.268113 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.269354 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.270395 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.270555 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:25.271501 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:25.271511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:25.271529 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:25.272184 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:25.272275 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:25.272304 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:25.272408 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:25.272581 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 05:23:25.272887 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":324,"threads_started":1}
I20260928 05:23:25.275508 14115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:25.276526 14115 log.cc:824] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:25.277525 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:23:25.279059 14115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e: No bootstrap required, opened a new log
I20260928 05:23:25.286245 14115 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } }
I20260928 05:23:25.286553 14115 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:25.286643 14115 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48e958b143e4e5a86b26256b1e9e33e, State: Initialized, Role: FOLLOWER
I20260928 05:23:25.287110 14115 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } }
I20260928 05:23:25.287307 14115 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:25.287423 14115 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:25.287561 14115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:25.289680 14115 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } }
I20260928 05:23:25.290122 14115 leader_election.cc:302] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48e958b143e4e5a86b26256b1e9e33e; no voters:
I20260928 05:23:25.290639 14115 leader_election.cc:288] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:25.290762 14120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:25.291404 14120 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [term 1 LEADER]: Becoming Leader. State: Replica: e48e958b143e4e5a86b26256b1e9e33e, State: Running, Role: LEADER
I20260928 05:23:25.291829 14120 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } }
I20260928 05:23:25.292378 14115 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:25.294266 14121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48e958b143e4e5a86b26256b1e9e33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } } }
I20260928 05:23:25.294474 14122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48e958b143e4e5a86b26256b1e9e33e. Latest consensus state: current_term: 1 leader_uuid: "e48e958b143e4e5a86b26256b1e9e33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48e958b143e4e5a86b26256b1e9e33e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46823 } } }
I20260928 05:23:25.294727 14121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:25.295307 14122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:25.295567 14127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:25.300485 14127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:25.309114 14127 catalog_manager.cc:1347] Generated new cluster ID: 7c46df475f294518aa4559b9adb92a97
I20260928 05:23:25.309212 14127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:25.334790 14127 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:25.335960 14127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:25.344493 14127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48e958b143e4e5a86b26256b1e9e33e: Generated new TSK 0
I20260928 05:23:25.345211 14127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:23:25.366581 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:25.358707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.8.124.254:46823)
0928 05:23:25.358874 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:25.358880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:25.359350 (+ 470us) server_negotiation.cc:407] Connection header received
0928 05:23:25.359527 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:25.359533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:25.359617 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:25.359774 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:23:25.360151 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.361235 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.362548 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.363231 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.363365 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:25.363787 (+ 422us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:23:25.365960 (+ 2173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:23:25.366277 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 05:23:25.366700 14118 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 05:23:25.368301 14144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:25.358714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46823 (local address 127.0.0.1:59656)
0928 05:23:25.359253 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:25.359269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:25.359378 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:25.359761 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:25.359765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:25.359773 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:23:25.360023 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:25.360030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.361359 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.361364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:25.362423 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:25.362436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.363400 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.363406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:25.363523 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:25.363554 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:23:25.366151 (+ 2597us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:23:25.367890 (+ 1739us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.124.254:46823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":1484,"thread_start_us":286,"threads_started":1}
I20260928 05:23:25.370105 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14049
I20260928 05:23:25.371168 14110 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 05:23:25.377694 14145 webserver.cc:533] Webserver: sslize failed:
E20260928 05:23:25.377779 14145 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:23:25.461468 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14049
2026-09-28T05:23:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (674 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T05:23:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:25Z Disabled control of system clock
I20260928 05:23:25.533524 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45297
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:39025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45297
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/testchainca.cert with env {}
W20260928 05:23:25.832450 14150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:25.832715 14150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:25.832819 14150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:25.839699 14150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:25.839790 14150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:25.839849 14150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:25.839919 14150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:25.839973 14150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:23:25.840054 14150 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:23:25.840106 14150 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 05:23:25.849426 14150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:25.850670 14150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:25.851846 14150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:25.862330 14155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:25.862378 14156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:25.862546 14158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:25.863310 14150 server_base.cc:1034] running on GCE node
I20260928 05:23:25.864001 14150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:25.865231 14150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:25.866448 14150 hybrid_clock.cc:648] HybridClock initialized: now 1790573005866395 us; error 45 us; skew 500 ppm
I20260928 05:23:25.866767 14150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:25.868873 14150 webserver.cc:460] Webserver started at http://127.8.124.254:34825/ using document root <none> and password file <none>
I20260928 05:23:25.869499 14150 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:25.869594 14150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:25.869901 14150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:25.872551 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "ef123da8af3f4a95b553e266b36adb9e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "7ee7638285c8ec327b3954c78cfecbc7"
server_key_iv: "48bd09e10aa0b30b201929fdc38e801f"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.873289 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "ef123da8af3f4a95b553e266b36adb9e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "7ee7638285c8ec327b3954c78cfecbc7"
server_key_iv: "48bd09e10aa0b30b201929fdc38e801f"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.878216 14150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:23:25.881243 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:25.882484 14150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:23:25.882630 14150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "ef123da8af3f4a95b553e266b36adb9e"
format_stamp: "Formatted at 2026-09-28 05:23:25 on dist-test-slave-515x"
server_key: "7ee7638285c8ec327b3954c78cfecbc7"
server_key_iv: "48bd09e10aa0b30b201929fdc38e801f"
server_key_version: "encryptionkey@0"
I20260928 05:23:25.882810 14150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:25.895403 14150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:25.896116 14150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:25.896356 14150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:25.912190 14150 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45297
I20260928 05:23:25.912210 14215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45297 every 8 connection(s)
I20260928 05:23:25.913407 14150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:25.916901 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:25.917549 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14150
I20260928 05:23:25.917735 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:25.918360 8691 external_mini_cluster.cc:1428] Setting key 54cd49a8afe2c61851137eeda6d4e1ed
I20260928 05:23:25.929997 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e: Bootstrap starting.
I20260928 05:23:25.932158 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:25.920166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.8.124.254:45297)
0928 05:23:25.921194 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:25.921207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:25.921259 (+ 52us) server_negotiation.cc:407] Connection header received
0928 05:23:25.922944 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:25.922970 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:25.923529 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:25.923934 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:25.924416 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.926196 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.927445 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:25.928431 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:25.928582 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:25.929588 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:25.929622 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:25.929639 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:23:25.930294 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:25.930393 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:23:25.930417 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:25.930527 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:25.930701 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 05:23:25.930979 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":293,"threads_started":1}
I20260928 05:23:25.933435 14216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:25.934584 14216 log.cc:824] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:25.935741 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:23:25.937036 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e: No bootstrap required, opened a new log
I20260928 05:23:25.943631 14216 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } }
I20260928 05:23:25.943938 14216 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:25.944016 14216 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef123da8af3f4a95b553e266b36adb9e, State: Initialized, Role: FOLLOWER
I20260928 05:23:25.944478 14216 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } }
I20260928 05:23:25.944667 14216 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:25.944775 14216 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:25.944917 14216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:25.947017 14216 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } }
I20260928 05:23:25.947443 14216 leader_election.cc:302] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef123da8af3f4a95b553e266b36adb9e; no voters:
I20260928 05:23:25.947949 14216 leader_election.cc:288] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:25.948068 14221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:25.948654 14221 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [term 1 LEADER]: Becoming Leader. State: Replica: ef123da8af3f4a95b553e266b36adb9e, State: Running, Role: LEADER
I20260928 05:23:25.949224 14221 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } }
I20260928 05:23:25.949929 14216 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:25.951704 14223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef123da8af3f4a95b553e266b36adb9e. Latest consensus state: current_term: 1 leader_uuid: "ef123da8af3f4a95b553e266b36adb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } } }
I20260928 05:23:25.952086 14223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:25.953725 14230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:25.953553 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef123da8af3f4a95b553e266b36adb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123da8af3f4a95b553e266b36adb9e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45297 } } }
I20260928 05:23:25.953859 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:25.958258 14230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:25.966694 14230 catalog_manager.cc:1347] Generated new cluster ID: 43eb9396315a40f587984c45a01ded60
I20260928 05:23:25.966781 14230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:25.980566 14230 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:25.981770 14230 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:25.992374 14230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef123da8af3f4a95b553e266b36adb9e: Generated new TSK 0
I20260928 05:23:25.993367 14230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:23:26.451251 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:26.010606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.8.124.254:45297)
0928 05:23:26.010801 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:26.010809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:26.011248 (+ 439us) server_negotiation.cc:407] Connection header received
0928 05:23:26.011456 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:26.011460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:26.011523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:26.011643 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:23:26.012142 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:26.012824 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:26.014826 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:26.015266 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:26.015357 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:26.015883 (+ 526us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:23:26.450666 (+434783us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:23:26.451025 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 05:23:26.451376 14219 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 05:23:26.451570 14245 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:26.010560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45297 (local address 127.0.0.1:49378)
0928 05:23:26.011147 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:26.011166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:26.011314 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:26.011674 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:26.011680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:26.011693 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:23:26.012006 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:26.012017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:26.012959 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:26.012966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:26.014692 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:26.014707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:26.015426 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:26.015433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:26.015622 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:26.015666 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:23:26.450950 (+435284us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:23:26.451222 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.124.254:45297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260928 05:23:26.452760 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14150
I20260928 05:23:26.453856 14211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:26.552445 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14150
2026-09-28T05:23:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1072 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (382 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (389 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:27 dist-test-slave-515x krb5kdc[14260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:27 dist-test-slave-515x krb5kdc[14260](info): set up 2 sockets
Sep 28 05:23:27 dist-test-slave-515x krb5kdc[14260](info): commencing operation
krb5kdc: starting...
W20260928 05:23:28.321970 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:28 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:28.499066 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37851
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:37851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:28.798779 14273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:28.799034 14273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:28.799120 14273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:28.806149 14273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:28.806241 14273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:28.806290 14273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:28.806353 14273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:28.806396 14273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:28.815649 14273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:37851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:28.816797 14273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:28.818028 14273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:28.829186 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:28.829272 14279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:28.829237 14278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:28.834944 14273 server_base.cc:1034] running on GCE node
I20260928 05:23:28.835659 14273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:28.836980 14273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:28.838167 14273 hybrid_clock.cc:648] HybridClock initialized: now 1790573008838123 us; error 32 us; skew 500 ppm
I20260928 05:23:28.838476 14273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:28 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573008, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:28.844094 14273 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:28.845460 14273 webserver.cc:460] Webserver started at http://127.8.124.254:45305/ using document root <none> and password file <none>
I20260928 05:23:28.846143 14273 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:28.846243 14273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:28.846534 14273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:28.849259 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "8d81acc95806437ba19c794220fc7ae1"
format_stamp: "Formatted at 2026-09-28 05:23:28 on dist-test-slave-515x"
server_key: "bda789bb3aeb28f3f3759f9025949c22"
server_key_iv: "20554366147844b0701ee5535593a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:28.849998 14273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "8d81acc95806437ba19c794220fc7ae1"
format_stamp: "Formatted at 2026-09-28 05:23:28 on dist-test-slave-515x"
server_key: "bda789bb3aeb28f3f3759f9025949c22"
server_key_iv: "20554366147844b0701ee5535593a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:28.854727 14273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:23:28.857864 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:28.859094 14273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:28.859228 14273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "8d81acc95806437ba19c794220fc7ae1"
format_stamp: "Formatted at 2026-09-28 05:23:28 on dist-test-slave-515x"
server_key: "bda789bb3aeb28f3f3759f9025949c22"
server_key_iv: "20554366147844b0701ee5535593a7f9"
server_key_version: "encryptionkey@0"
I20260928 05:23:28.859413 14273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:28.876039 14273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:28.879592 14273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:28.879815 14273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:28.895754 14273 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:37851
I20260928 05:23:28.895783 14339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:37851 every 8 connection(s)
I20260928 05:23:28.896920 14273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:28.900481 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:28.905673 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14273
I20260928 05:23:28.905871 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:28.906639 8691 external_mini_cluster.cc:1428] Setting key 978da39110c102d9d95fb5ba0fbeb608
I20260928 05:23:28.911144 14340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1: Bootstrap starting.
I20260928 05:23:28.914466 14340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:28.915587 14340 log.cc:824] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:28.918448 14340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1: No bootstrap required, opened a new log
Sep 28 05:23:28 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:28.927892 14340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } }
I20260928 05:23:28.928305 14340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:28.928413 14340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d81acc95806437ba19c794220fc7ae1, State: Initialized, Role: FOLLOWER
I20260928 05:23:28.929049 14340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } }
I20260928 05:23:28.929291 14340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:28.929471 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:28.908600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.8.124.254:37851)
0928 05:23:28.909394 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:28.909405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:28.909446 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:23:28.911045 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:28.911066 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:28.911452 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:28.911750 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:28.913362 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:28.914674 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:28.916386 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:28.917129 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:28.917248 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:28.921460 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:28.921470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:28.921488 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:28.921536 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:28.925338 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:28.925992 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:28.926000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:28.926008 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:28.926198 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:28.926542 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:28.926550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:28.926553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:28.927184 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:28.927428 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:28.927741 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 05:23:28.928204 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":401,"threads_started":1}
I20260928 05:23:28.929438 14340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:28.929682 14340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:28.931834 14340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } }
I20260928 05:23:28.932268 14340 leader_election.cc:302] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d81acc95806437ba19c794220fc7ae1; no voters:
I20260928 05:23:28.932773 14340 leader_election.cc:288] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:28.932878 14345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:28.933439 14345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [term 1 LEADER]: Becoming Leader. State: Replica: 8d81acc95806437ba19c794220fc7ae1, State: Running, Role: LEADER
I20260928 05:23:28.933899 14345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } }
I20260928 05:23:28.934659 14340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:28.936326 14346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d81acc95806437ba19c794220fc7ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } } }
I20260928 05:23:28.936678 14346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:28.937312 14347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d81acc95806437ba19c794220fc7ae1. Latest consensus state: current_term: 1 leader_uuid: "8d81acc95806437ba19c794220fc7ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d81acc95806437ba19c794220fc7ae1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37851 } } }
I20260928 05:23:28.937680 14347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:28.938159 14352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:28.942528 14352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:28.951551 14352 catalog_manager.cc:1347] Generated new cluster ID: cbdb0e33764c41ab820dc7f727960995
I20260928 05:23:28.951648 14352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:28.969732 14352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:28.971452 14352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:28.980111 14352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d81acc95806437ba19c794220fc7ae1: Generated new TSK 0
I20260928 05:23:28.981456 14352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:29.037675 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:29.348909 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:29.349166 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:29.349294 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:29.356191 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:29.356302 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:29.356451 14366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:29.365716 14366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:29.366878 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:29.368093 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:29.379745 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:29.379679 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:29.379716 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:29.380653 14366 server_base.cc:1034] running on GCE node
I20260928 05:23:29.381139 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:29.381981 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:29.383170 14366 hybrid_clock.cc:648] HybridClock initialized: now 1790573009383119 us; error 43 us; skew 500 ppm
I20260928 05:23:29.383451 14366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:29 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:29.386181 14366 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:29.387460 14366 webserver.cc:460] Webserver started at http://127.8.124.193:38035/ using document root <none> and password file <none>
I20260928 05:23:29.388156 14366 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:29.388267 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:29.388538 14366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:29.391295 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "5dedf246c273454cabcd3b54a4b33b45"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "f9ca25594a5cf0f57f3b2c07b3f20c4a"
server_key_iv: "4feda36bcdecd15da4913694c118d2cf"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.392015 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "5dedf246c273454cabcd3b54a4b33b45"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "f9ca25594a5cf0f57f3b2c07b3f20c4a"
server_key_iv: "4feda36bcdecd15da4913694c118d2cf"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.396649 14366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:23:29.399791 14381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.401028 14366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:23:29.401165 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "5dedf246c273454cabcd3b54a4b33b45"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "f9ca25594a5cf0f57f3b2c07b3f20c4a"
server_key_iv: "4feda36bcdecd15da4913694c118d2cf"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.401331 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:29.417282 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:29.420655 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:29.420897 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:29.421783 14366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:29.422976 14366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:29.423045 14366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.423156 14366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:29.423209 14366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.451015 14366 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:34561
I20260928 05:23:29.451056 14493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:34561 every 8 connection(s)
I20260928 05:23:29.452170 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:29.452726 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14366
I20260928 05:23:29.452911 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:29.453685 8691 external_mini_cluster.cc:1428] Setting key d3e00f736076dadf5511062d99d82660
Sep 28 05:23:29 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:29.472262 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:29.454711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:41887 (local address 127.8.124.254:37851)
0928 05:23:29.455123 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:29.455129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:29.456420 (+ 1291us) server_negotiation.cc:407] Connection header received
0928 05:23:29.458421 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:29.458425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:29.458486 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:29.458574 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:29.460310 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.460984 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.462930 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.463382 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.463485 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:29.466807 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:29.466811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:29.466815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:29.466844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:29.469100 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:29.470003 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:29.470007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:29.470010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:29.470134 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:29.470632 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:29.470635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:29.470638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:29.470907 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:29.471013 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:29.471844 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 05:23:29.472013 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:23:29.473541 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:29.455179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37851 (local address 127.8.124.193:41887)
0928 05:23:29.455963 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:29.456009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:29.458161 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:29.458804 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:29.458814 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:29.459442 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:29.460152 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:29.460175 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.461128 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.461132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:29.462797 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:29.462810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.463591 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.463598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:29.463830 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:29.464418 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:29.464440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:29.466609 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:29.469226 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:29.469235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:29.469253 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:29.469874 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:29.470235 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:29.470239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:29.470243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:29.470496 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:29.471022 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:29.471030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:29.471451 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 05:23:29.471751 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 05:23:29.472273 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":315,"threads_started":1}
I20260928 05:23:29.474895 14494 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37851
I20260928 05:23:29.475216 14494 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:23:29.476068 14494 heartbeater.cc:511] Master 127.8.124.254:37851 requested a full tablet report, sending...
I20260928 05:23:29.478164 14305 ts_manager.cc:194] Registered new tserver with Master: 5dedf246c273454cabcd3b54a4b33b45 (127.8.124.193:34561)
I20260928 05:23:29.479671 14305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:41887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:29.496796 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:29.795414 14500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:29.795684 14500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:29.795801 14500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:29.802779 14500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:29.802878 14500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:29.803027 14500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:29.812417 14500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:29.813645 14500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:29.814874 14500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:29.826886 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:29.826925 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:29.827069 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:29.827643 14500 server_base.cc:1034] running on GCE node
I20260928 05:23:29.828151 14500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:29.828996 14500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:29.830178 14500 hybrid_clock.cc:648] HybridClock initialized: now 1790573009830132 us; error 33 us; skew 500 ppm
I20260928 05:23:29.830456 14500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:29 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:29.833252 14500 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:29.834589 14500 webserver.cc:460] Webserver started at http://127.8.124.194:38593/ using document root <none> and password file <none>
I20260928 05:23:29.835220 14500 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:29.835330 14500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:29.835602 14500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:29.838339 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "9ef45bfa9c0d4750831b5cf61141c05d"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "fca9b273552dc92c20bd01c32aef35f0"
server_key_iv: "e1d66cec4319fc983fc1f54bb6f0da02"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.839049 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef45bfa9c0d4750831b5cf61141c05d"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "fca9b273552dc92c20bd01c32aef35f0"
server_key_iv: "e1d66cec4319fc983fc1f54bb6f0da02"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.843721 14500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:23:29.846894 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.848129 14500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:29.848269 14500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "9ef45bfa9c0d4750831b5cf61141c05d"
format_stamp: "Formatted at 2026-09-28 05:23:29 on dist-test-slave-515x"
server_key: "fca9b273552dc92c20bd01c32aef35f0"
server_key_iv: "e1d66cec4319fc983fc1f54bb6f0da02"
server_key_version: "encryptionkey@0"
I20260928 05:23:29.848448 14500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:29.866294 14500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:29.870172 14500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:29.870420 14500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:29.871289 14500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:29.872532 14500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:29.872604 14500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.872718 14500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:29.872766 14500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:29.901391 14500 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:33401
I20260928 05:23:29.901392 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:33401 every 8 connection(s)
I20260928 05:23:29.902835 14500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:29.912388 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14500
I20260928 05:23:29.912547 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:29.913079 8691 external_mini_cluster.cc:1428] Setting key d68398597f07e3060a972be900c51fda
Sep 28 05:23:29 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573009, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:29.922528 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:29.905491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57639 (local address 127.8.124.254:37851)
0928 05:23:29.905668 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:29.905673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:29.906901 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 05:23:29.908849 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:29.908854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:29.908958 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:29.909044 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:29.910518 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.911229 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.913015 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.913550 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.913702 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:29.916969 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:29.916974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:29.916979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:29.917012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:29.919785 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:29.920430 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:29.920435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:29.920438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:29.920576 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:29.920946 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:29.920950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:29.920952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:29.921196 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:29.921320 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:29.922007 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 05:23:29.922213 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 05:23:29.923449 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:29.905984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37851 (local address 127.8.124.194:57639)
0928 05:23:29.906725 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:29.906764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:29.908610 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:29.909272 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:29.909280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:29.909804 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:29.910366 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:29.910384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.911372 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.911375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:29.912878 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:29.912893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:29.913785 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:29.913792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:29.914005 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:29.914609 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:29.914633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:29.916773 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:29.919917 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:29.919923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:29.919934 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:29.920308 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:29.920676 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:29.920679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:29.920681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:29.920839 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:29.921313 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:29.921321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:29.921666 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 05:23:29.921898 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 05:23:29.922374 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":322,"threads_started":1}
I20260928 05:23:29.924911 14628 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37851
I20260928 05:23:29.925215 14628 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:29.926033 14628 heartbeater.cc:511] Master 127.8.124.254:37851 requested a full tablet report, sending...
I20260928 05:23:29.927511 14305 ts_manager.cc:194] Registered new tserver with Master: 9ef45bfa9c0d4750831b5cf61141c05d (127.8.124.194:33401)
I20260928 05:23:29.928570 14305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:57639
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:29.957379 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:30.261377 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:30.261660 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:30.261793 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:30.268724 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:30.268821 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:30.268956 14633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:30.279424 14633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:30.280588 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:30.281829 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:30.293525 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:30.293525 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:30.293742 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:30.293910 14633 server_base.cc:1034] running on GCE node
I20260928 05:23:30.294556 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:30.295369 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:30.296614 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790573010296562 us; error 40 us; skew 500 ppm
I20260928 05:23:30.296921 14633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.299782 14633 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:30.301041 14633 webserver.cc:460] Webserver started at http://127.8.124.195:38495/ using document root <none> and password file <none>
I20260928 05:23:30.301720 14633 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:30.301831 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:30.302101 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:30.304888 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "a6dad05c069343fe9657176306a72222"
format_stamp: "Formatted at 2026-09-28 05:23:30 on dist-test-slave-515x"
server_key: "18bccb068e6649a0fee3c24ebb2995aa"
server_key_iv: "bc079df83bf248309b52b6b96f93ed7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:30.305594 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "a6dad05c069343fe9657176306a72222"
format_stamp: "Formatted at 2026-09-28 05:23:30 on dist-test-slave-515x"
server_key: "18bccb068e6649a0fee3c24ebb2995aa"
server_key_iv: "bc079df83bf248309b52b6b96f93ed7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:30.310200 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:23:30.313190 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:30.314474 14633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:30.314611 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "a6dad05c069343fe9657176306a72222"
format_stamp: "Formatted at 2026-09-28 05:23:30 on dist-test-slave-515x"
server_key: "18bccb068e6649a0fee3c24ebb2995aa"
server_key_iv: "bc079df83bf248309b52b6b96f93ed7e"
server_key_version: "encryptionkey@0"
I20260928 05:23:30.314779 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:30.332429 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:30.336232 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:30.336483 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:30.337349 14633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:30.338573 14633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:30.338649 14633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:30.338768 14633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:30.338819 14633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:30.366624 14633 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:37901
I20260928 05:23:30.366659 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:37901 every 8 connection(s)
I20260928 05:23:30.367772 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:30.374114 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14633
I20260928 05:23:30.374258 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:30.374728 8691 external_mini_cluster.cc:1428] Setting key 3296e12ca44c638ad4c9e8649103bf80
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:30.385689 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.369937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47803 (local address 127.8.124.254:37851)
0928 05:23:30.370130 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.370136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.371315 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 05:23:30.372936 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.372940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.373003 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.373085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.374556 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.375500 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.377348 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.377820 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.377938 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.380820 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.380825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.380828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.380856 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.383212 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.383819 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.383824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.383827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.383970 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.384305 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.384308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.384310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.384553 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.384657 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.385225 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.385412 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:23:30.386483 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.370489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37851 (local address 127.8.124.195:47803)
0928 05:23:30.371161 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:30.371199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:30.372710 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:30.373262 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:30.373270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:30.373824 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:30.374395 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:30.374418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.375643 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.375647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:30.377207 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:30.377222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.378020 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.378026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:30.378204 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.378727 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:30.378751 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:30.380644 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:30.383307 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:30.383313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:30.383323 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:30.383683 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:30.384068 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:30.384071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:30.384073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:30.384225 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:30.384641 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:30.384646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:30.384918 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 05:23:30.385120 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 05:23:30.385445 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260928 05:23:30.387677 14761 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37851
I20260928 05:23:30.387962 14761 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:30.388734 14761 heartbeater.cc:511] Master 127.8.124.254:37851 requested a full tablet report, sending...
I20260928 05:23:30.390074 14305 ts_manager.cc:194] Registered new tserver with Master: a6dad05c069343fe9657176306a72222 (127.8.124.195:37901)
I20260928 05:23:30.390795 14305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:47803
I20260928 05:23:30.401280 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/test-admin.keytab.
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.424808 8691 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 05:23:30.429150 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:30.427145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.8.124.193:34561)
0928 05:23:30.427676 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.427683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.427709 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:23:30.427808 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.427815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.428021 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.428147 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.428855 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51640: BlockingRecv error: recv got EOF from 127.0.0.1:51640 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":310,"threads_started":1}
W20260928 05:23:30.429780 8691 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.431836 8691 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.433475 8691 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:30.448558 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.435447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.8.124.193:34561)
0928 05:23:30.435606 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.435611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.435816 (+ 205us) server_negotiation.cc:407] Connection header received
0928 05:23:30.435961 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.435967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.436037 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.436132 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.437277 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.438588 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.439802 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.440329 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.440445 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.443754 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.443764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.443779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.443819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.446271 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.446866 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.446872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.446875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.447003 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.447293 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.447296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.447298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.447615 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.447818 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.448061 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.448276 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 05:23:30.454172 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:30.452565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.8.124.193:34561)
0928 05:23:30.452727 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.452732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.453043 (+ 311us) server_negotiation.cc:407] Connection header received
0928 05:23:30.453201 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.453206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.453270 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.453341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.453985 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51654: BlockingRecv error: recv got EOF from 127.0.0.1:51654 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 05:23:30.454713 8691 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.456297 8691 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.457698 8691 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:30.470228 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.459130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.8.124.193:34561)
0928 05:23:30.459296 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.459301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.459593 (+ 292us) server_negotiation.cc:407] Connection header received
0928 05:23:30.459722 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.459725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.459773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.459833 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.460716 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.461377 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.462619 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.463002 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.463090 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.466111 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.466116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.466119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.466146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.468301 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.468818 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.468822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.468824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.468923 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.469192 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.469195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.469197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.469435 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.469614 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.469825 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.469970 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 05:23:30.473905 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:30.472399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.8.124.193:34561)
0928 05:23:30.472560 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.472566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.472784 (+ 218us) server_negotiation.cc:407] Connection header received
0928 05:23:30.472918 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.472923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.472990 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.473065 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.473687 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51666: BlockingRecv error: recv got EOF from 127.0.0.1:51666 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 05:23:30.474041 8691 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572965871275-8691-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.475340 8691 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:30.476461 8691 init.cc:295] Successfully reacquired a new kerberos TGT
I20260928 05:23:30.482739 14494 heartbeater.cc:503] Master 127.8.124.254:37851 was elected leader, sending a full tablet report...
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:30.489002 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.477550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.8.124.193:34561)
0928 05:23:30.477737 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.477743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.477933 (+ 190us) server_negotiation.cc:407] Connection header received
0928 05:23:30.478036 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.478039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.478087 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.478155 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.478938 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.479605 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.480702 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.481084 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.481164 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.484473 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.484478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.484481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.484508 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.486794 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.487331 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.487336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.487339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.487486 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.487799 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.487804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.487806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.488054 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.488306 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.488550 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.488736 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:23:30.490301 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14366
I20260928 05:23:30.499917 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.490892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.8.124.193:34561)
0928 05:23:30.491006 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.491010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.491316 (+ 306us) server_negotiation.cc:407] Connection header received
0928 05:23:30.491429 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.491432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.491477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.491543 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.492236 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.492864 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.493974 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.494363 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.494454 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.496085 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.496089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.496092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.496112 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.498143 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.498644 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.498648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.498650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.498761 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.499003 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.499006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.499008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.499202 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.499363 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.499572 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.499707 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:23:30.500346 14489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:30.580430 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14366
I20260928 05:23:30.593492 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14500
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:30.607707 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.594365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.8.124.194:33401)
0928 05:23:30.594894 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.594901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.594925 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:30.594984 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.594990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.595188 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.595313 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.596269 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.597276 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.598564 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.599090 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.599178 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.602702 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.602712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.602726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.602759 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.605303 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.606023 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.606030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.606032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.606147 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.606431 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.606433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.606435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.606728 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.606954 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.607239 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.607433 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260928 05:23:30.608865 14623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:30.689198 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14500
I20260928 05:23:30.701454 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14633
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:30.715569 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:30.702386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.8.124.195:37901)
0928 05:23:30.702981 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:30.702988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:30.703017 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:23:30.703076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:30.703083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:30.703283 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:30.703423 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:30.704337 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.705382 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.706622 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:30.707222 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:30.707315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:30.710884 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:30.710894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:30.710908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:30.710938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:30.713311 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.713911 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.713918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.713920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.714028 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:30.714336 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:30.714340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:30.714342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:30.714647 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:30.714855 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:30.715070 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 05:23:30.715278 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
I20260928 05:23:30.716745 14756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:30.796344 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14633
I20260928 05:23:30.808427 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14273
I20260928 05:23:30.809545 14335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:30.906957 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14273
2026-09-28T05:23:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3585 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14788](info): set up 2 sockets
Sep 28 05:23:30 dist-test-slave-515x krb5kdc[14788](info): commencing operation
krb5kdc: starting...
W20260928 05:23:31.889936 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:32 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab.
Entry for principal oryx/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab.
I20260928 05:23:32.074960 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36501
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab
--principal=oryx/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:32.389640 14801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:32.389904 14801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:32.390015 14801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:32.397003 14801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:32.397101 14801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:32.397171 14801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:32.397245 14801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:32.397301 14801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:32.406730 14801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab
--principal=oryx/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:32.407966 14801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:32.409209 14801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:32.419677 14806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:32.419709 14807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:32.419751 14809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:32.420636 14801 server_base.cc:1034] running on GCE node
I20260928 05:23:32.421362 14801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:32.422668 14801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:32.423858 14801 hybrid_clock.cc:648] HybridClock initialized: now 1790573012423805 us; error 43 us; skew 500 ppm
I20260928 05:23:32.424158 14801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:32 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:32.426995 14801 init.cc:377] Logged in from keytab as oryx/127.8.124.254@KRBTEST.COM (short username oryx)
I20260928 05:23:32.428306 14801 webserver.cc:460] Webserver started at http://127.8.124.254:37543/ using document root <none> and password file <none>
I20260928 05:23:32.428920 14801 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:32.429030 14801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:32.429283 14801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:32.432089 14801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "81e8f1656733456a9c4d3d123be61db1"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "e500324af2ddba9005438a364b0ddb2a"
server_key_iv: "7ff0b6e8608d510f77b5c447a0ea1460"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.432757 14801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "81e8f1656733456a9c4d3d123be61db1"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "e500324af2ddba9005438a364b0ddb2a"
server_key_iv: "7ff0b6e8608d510f77b5c447a0ea1460"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.437321 14801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:23:32.440361 14816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:32.441563 14801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:32.441740 14801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "81e8f1656733456a9c4d3d123be61db1"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "e500324af2ddba9005438a364b0ddb2a"
server_key_iv: "7ff0b6e8608d510f77b5c447a0ea1460"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.441892 14801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:32.455255 14801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:32.458693 14801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:32.458921 14801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:32.475135 14801 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:36501
I20260928 05:23:32.475162 14867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:36501 every 8 connection(s)
I20260928 05:23:32.476366 14801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:32.479795 14868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:32.482429 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14801
I20260928 05:23:32.482615 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:32.483349 8691 external_mini_cluster.cc:1428] Setting key cf2a1860d8f790ba2f69a01c6127f100
I20260928 05:23:32.491381 14868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Bootstrap starting.
I20260928 05:23:32.494800 14868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:32.495988 14868 log.cc:824] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:23:32 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:32.498826 14868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: No bootstrap required, opened a new log
I20260928 05:23:32.504631 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:32.485259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.8.124.254:36501)
0928 05:23:32.485975 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:32.485986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:32.486026 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:23:32.487622 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:32.487643 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:32.488026 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:32.488366 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:32.489998 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:32.491237 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:32.492889 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:32.493520 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:32.493673 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:32.498195 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:32.498202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:32.498214 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:32.498247 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:32.501001 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:32.501723 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:32.501729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:32.501735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:32.501901 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:32.502249 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:32.502252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:32.502254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:32.502725 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:32.502961 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:32.503356 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:23:32.503678 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":337,"threads_started":1}
I20260928 05:23:32.506796 14868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } }
I20260928 05:23:32.507117 14868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:32.507210 14868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e8f1656733456a9c4d3d123be61db1, State: Initialized, Role: FOLLOWER
I20260928 05:23:32.507714 14868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } }
I20260928 05:23:32.507912 14868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:32.508025 14868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:32.508160 14868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:32.510162 14868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } }
I20260928 05:23:32.510622 14868 leader_election.cc:302] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e8f1656733456a9c4d3d123be61db1; no voters:
I20260928 05:23:32.511222 14868 leader_election.cc:288] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:32.511298 14873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:32.511893 14873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 LEADER]: Becoming Leader. State: Replica: 81e8f1656733456a9c4d3d123be61db1, State: Running, Role: LEADER
I20260928 05:23:32.512382 14873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } }
I20260928 05:23:32.512993 14868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:32.514787 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } }
I20260928 05:23:32.515112 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:32.518008 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e8f1656733456a9c4d3d123be61db1. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } }
I20260928 05:23:32.518390 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:32.518784 14880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:32.523833 14880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:32.531920 14880 catalog_manager.cc:1347] Generated new cluster ID: 5e13c503a25f44569fb8ec805868a783
I20260928 05:23:32.532023 14880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:32.544698 14880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:32.546165 14880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:32.557942 14880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Generated new TSK 0
I20260928 05:23:32.558913 14880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
I20260928 05:23:32.607475 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:32.913384 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:32.913676 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:32.913805 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:32.920727 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:32.920830 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:32.920975 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:32.930159 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:32.931532 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:32.932813 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:32.944938 14900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:32.945013 14902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:32.945209 14899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:32.945974 14894 server_base.cc:1034] running on GCE node
I20260928 05:23:32.946532 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:32.947381 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:32.948567 14894 hybrid_clock.cc:648] HybridClock initialized: now 1790573012948518 us; error 39 us; skew 500 ppm
I20260928 05:23:32.948846 14894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:32 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:32.951813 14894 init.cc:377] Logged in from keytab as oryx/127.8.124.193@KRBTEST.COM (short username oryx)
I20260928 05:23:32.953164 14894 webserver.cc:460] Webserver started at http://127.8.124.193:44847/ using document root <none> and password file <none>
I20260928 05:23:32.953922 14894 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:32.954046 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:32.954317 14894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:32.957160 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "3bfe8e2ab3862e2796cca00e82c5c012"
server_key_iv: "b9c10bdfb1c7e870847f0589f8bc57d5"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.957885 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "3bfe8e2ab3862e2796cca00e82c5c012"
server_key_iv: "b9c10bdfb1c7e870847f0589f8bc57d5"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.962687 14894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:23:32.966005 14909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:32.967224 14894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:23:32.967361 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "3bfe8e2ab3862e2796cca00e82c5c012"
server_key_iv: "b9c10bdfb1c7e870847f0589f8bc57d5"
server_key_version: "encryptionkey@0"
I20260928 05:23:32.967540 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:32.980633 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:32.984077 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:32.984326 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:32.985158 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:32.986456 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:32.986522 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:32.986631 14894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:32.986685 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.015288 14894 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33569
I20260928 05:23:33.015323 15021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33569 every 8 connection(s)
I20260928 05:23:33.016525 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:33.026647 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 14894
I20260928 05:23:33.026834 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:33.027446 8691 external_mini_cluster.cc:1428] Setting key 11d4a40099ac040dbce68a24a8efea38
Sep 28 05:23:33 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:33.035159 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.018844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:36231 (local address 127.8.124.254:36501)
0928 05:23:33.019225 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:33.019230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:33.020160 (+ 930us) server_negotiation.cc:407] Connection header received
0928 05:23:33.021986 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:33.021991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:33.022064 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:33.022156 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:33.023575 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.024274 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.025752 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.026154 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.026263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.029476 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:33.029482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:33.029487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:33.029520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:33.032395 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.033006 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.033011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.033014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.033137 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.033489 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.033493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.033496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.033818 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:33.033947 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:33.034755 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 05:23:33.034917 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":202,"threads_started":1}
I20260928 05:23:33.036249 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.019327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:36231)
0928 05:23:33.020006 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:33.020044 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:33.021730 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:33.022379 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:33.022395 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:33.022858 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:33.023416 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.023434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.024409 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.024413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.025632 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.025646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.026354 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.026361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.026586 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.027196 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:33.027217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:33.029280 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:33.032525 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.032531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.032543 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.032914 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.033229 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.033232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.033234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.033402 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.034011 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:33.034018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:33.034407 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 05:23:33.034662 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 05:23:33.035117 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260928 05:23:33.037973 15022 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:33.038277 15022 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:33.039023 15022 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:33.041034 14833 ts_manager.cc:194] Registered new tserver with Master: a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569)
I20260928 05:23:33.042435 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:36231
WARNING: no policy specified for oryx/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
I20260928 05:23:33.070142 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:33.372123 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:33.372419 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:33.372558 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:33.379494 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:33.379606 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:33.379755 15028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:33.388952 15028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:33.390144 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:33.391376 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:33.403301 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:33.403258 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:33.403474 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:33.403859 15028 server_base.cc:1034] running on GCE node
I20260928 05:23:33.404474 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:33.405332 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:33.406538 15028 hybrid_clock.cc:648] HybridClock initialized: now 1790573013406480 us; error 47 us; skew 500 ppm
I20260928 05:23:33.406833 15028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:33 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573013, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:33.409767 15028 init.cc:377] Logged in from keytab as oryx/127.8.124.194@KRBTEST.COM (short username oryx)
I20260928 05:23:33.411123 15028 webserver.cc:460] Webserver started at http://127.8.124.194:35219/ using document root <none> and password file <none>
I20260928 05:23:33.411791 15028 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:33.411906 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:33.412166 15028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:33.414948 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "23f5377cb1844d3980f2fa588288120a"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "31d26d7024830ea13122f21cb6742289"
server_key_iv: "fcbb8e20fa27dab57aeb4a4b56996794"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.415637 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "23f5377cb1844d3980f2fa588288120a"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "31d26d7024830ea13122f21cb6742289"
server_key_iv: "fcbb8e20fa27dab57aeb4a4b56996794"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.420254 15028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:33.423424 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.424615 15028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:33.424751 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "23f5377cb1844d3980f2fa588288120a"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "31d26d7024830ea13122f21cb6742289"
server_key_iv: "fcbb8e20fa27dab57aeb4a4b56996794"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.424917 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:33.446081 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:33.449810 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:33.450055 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:33.450951 15028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:33.452157 15028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:33.452235 15028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.452356 15028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:33.452409 15028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.480805 15028 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:37565
I20260928 05:23:33.480845 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:37565 every 8 connection(s)
I20260928 05:23:33.482012 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:33.486690 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15028
I20260928 05:23:33.486855 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:33.487388 8691 external_mini_cluster.cc:1428] Setting key 1bf8475a0ea9248b1b08d8369c5e08a3
Sep 28 05:23:33 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573013, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:33.501848 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.484422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:33261 (local address 127.8.124.254:36501)
0928 05:23:33.484611 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:33.484617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:33.485822 (+ 1205us) server_negotiation.cc:407] Connection header received
0928 05:23:33.488125 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:33.488130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:33.488208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:33.488288 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:33.489967 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.490827 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.492730 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.493121 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.493206 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.496829 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:33.496834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:33.496838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:33.496866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:33.499145 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.499915 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.499918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.499920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.500045 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.500439 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.500442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.500444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.500658 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:33.500759 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:33.501458 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 05:23:33.501635 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:23:33.503175 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.484904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:33261)
0928 05:23:33.485666 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:33.485709 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:33.487846 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:33.488533 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:33.488542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:33.489056 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:33.489784 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.489819 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.490980 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.490985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.492614 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.492627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.493301 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.493307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.493552 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.494214 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:33.494240 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:33.496630 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:33.499270 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.499278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.499292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.499811 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.500130 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.500134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.500137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.500353 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.500754 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:33.500761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:33.501116 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 05:23:33.501399 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 05:23:33.501880 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":293,"threads_started":1}
I20260928 05:23:33.504546 15156 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:33.504810 15156 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:33.505436 15156 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:33.506733 14833 ts_manager.cc:194] Registered new tserver with Master: 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
WARNING: no policy specified for oryx/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:23:33.507494 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:33261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
I20260928 05:23:33.530018 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:33.832379 15161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:33.832633 15161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:33.832720 15161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:33.839659 15161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:33.839753 15161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:33.839860 15161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:33.849143 15161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:33.850353 15161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:33.851572 15161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:33.863463 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:33.863536 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:33.863731 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:33.864512 15161 server_base.cc:1034] running on GCE node
I20260928 05:23:33.865059 15161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:33.865983 15161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:33.867187 15161 hybrid_clock.cc:648] HybridClock initialized: now 1790573013867148 us; error 27 us; skew 500 ppm
I20260928 05:23:33.867496 15161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:33 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573013, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:33.874240 15161 init.cc:377] Logged in from keytab as oryx/127.8.124.195@KRBTEST.COM (short username oryx)
I20260928 05:23:33.875607 15161 webserver.cc:460] Webserver started at http://127.8.124.195:36861/ using document root <none> and password file <none>
I20260928 05:23:33.876260 15161 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:33.876369 15161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:33.876636 15161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:33.879433 15161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "2dcf3e16be9fff4d3da11b920d15f31f"
server_key_iv: "2c4fcec68159c67be3bbeb5bab04724a"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.880152 15161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "2dcf3e16be9fff4d3da11b920d15f31f"
server_key_iv: "2c4fcec68159c67be3bbeb5bab04724a"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.884886 15161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:33.888088 15176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.889391 15161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:33.889530 15161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "2dcf3e16be9fff4d3da11b920d15f31f"
server_key_iv: "2c4fcec68159c67be3bbeb5bab04724a"
server_key_version: "encryptionkey@0"
I20260928 05:23:33.889726 15161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:33.903000 15161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:33.906728 15161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:33.906985 15161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:33.907835 15161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:33.909025 15161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:33.909097 15161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.909207 15161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:33.909269 15161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:33.937107 15161 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:38645
I20260928 05:23:33.937139 15288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:38645 every 8 connection(s)
I20260928 05:23:33.938314 15161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:33.946853 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15161
I20260928 05:23:33.947014 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:33.947408 8691 external_mini_cluster.cc:1428] Setting key 07e5143c94b5d567178b31b8273fd935
Sep 28 05:23:33 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573013, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:33.956252 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.940719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:52459 (local address 127.8.124.254:36501)
0928 05:23:33.940883 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:33.940887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:33.942034 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 05:23:33.943733 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:33.943738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:33.943814 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:33.943904 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:33.945315 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.946109 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.947615 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.948023 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.948126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.951339 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:33.951344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:33.951347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:33.951377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:33.953519 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.954149 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.954154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.954156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.954271 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:33.954650 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:33.954653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:33.954655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:33.954955 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:33.955086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:33.955725 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 05:23:33.955946 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:23:33.957098 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:33.941214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:52459)
0928 05:23:33.941896 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:33.941931 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:33.943479 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:33.944120 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:33.944128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:33.944628 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:33.945161 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.945178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.946244 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.946247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.947496 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:33.947509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:33.948222 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:33.948228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:33.948459 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:33.949061 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:33.949082 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:33.951159 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:33.953659 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.953667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.953683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.954049 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.954365 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:33.954370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:33.954374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:33.954545 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:33.955080 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:33.955087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:33.955411 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 05:23:33.955622 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 05:23:33.956084 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":265,"threads_started":1}
I20260928 05:23:33.958299 15289 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:33.958555 15289 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:33.959163 15289 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:33.960420 14833 ts_manager.cc:194] Registered new tserver with Master: ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:33.961230 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:52459
I20260928 05:23:33.962041 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.124.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.253.keytab.
Entry for principal oryx/127.8.124.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.253.keytab.
I20260928 05:23:34.003176 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:46597
--webserver_interface=127.8.124.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.253.keytab
--principal=oryx/127.8.124.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36501,127.8.124.253:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:23:34.045660 15022 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
W20260928 05:23:34.306069 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:34.306324 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:34.306438 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:34.313402 15294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:34.313510 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:34.313583 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:34.313680 15294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:34.313733 15294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:34.323035 15294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36501,127.8.124.253:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.253.keytab
--principal=oryx/127.8.124.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.124.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
W20260928 05:23:34.323348 15294 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.124.254:36501,127.8.124.253:46597'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 05:23:34.324313 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:34.325587 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:34.336318 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.336279 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.336347 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:34.337121 15294 server_base.cc:1034] running on GCE node
I20260928 05:23:34.337843 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:34.339063 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:34.340263 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790573014340209 us; error 32 us; skew 500 ppm
I20260928 05:23:34.340544 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:34.343410 15294 init.cc:377] Logged in from keytab as oryx/127.8.124.253@KRBTEST.COM (short username oryx)
I20260928 05:23:34.344728 15294 webserver.cc:460] Webserver started at http://127.8.124.253:37323/ using document root <none> and password file <none>
I20260928 05:23:34.345355 15294 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:34.345434 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:34.345755 15294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:34.348410 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/instance:
uuid: "4d6bb68f784046f990e6314c002b855e"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "53fac1cfcca3b0128086c105372170c4"
server_key_iv: "e9d3d0f55026e32cdd44e7582d850054"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.349074 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal/instance:
uuid: "4d6bb68f784046f990e6314c002b855e"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "53fac1cfcca3b0128086c105372170c4"
server_key_iv: "e9d3d0f55026e32cdd44e7582d850054"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.353863 15294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260928 05:23:34.356993 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:34.358249 15294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:34.358434 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "4d6bb68f784046f990e6314c002b855e"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "53fac1cfcca3b0128086c105372170c4"
server_key_iv: "e9d3d0f55026e32cdd44e7582d850054"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.358608 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:34.372756 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:34.376600 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:34.376817 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:34.393137 15294 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.253:46597
I20260928 05:23:34.393157 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.253:46597 every 8 connection(s)
I20260928 05:23:34.394428 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
I20260928 05:23:34.397802 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:34.400440 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15294
I20260928 05:23:34.405032 15361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.253@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:34.422293 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.407805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.8.124.254:36501)
0928 05:23:34.407964 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.407968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.408663 (+ 695us) server_negotiation.cc:407] Connection header received
0928 05:23:34.408812 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.408818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.408885 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.408965 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.410197 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.410961 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.412646 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.413060 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.413153 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.416540 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.416544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.416547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.416578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.419484 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.420163 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.420166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.420169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.420285 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.420630 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.420634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.420637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.421032 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.421172 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.421800 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.421999 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:23:34.423142 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.402462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.8.124.253:46597)
0928 05:23:34.403187 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.403201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.403247 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:23:34.405304 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.405330 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.405866 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.406275 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.407495 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.409258 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.410804 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.411551 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.411666 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.416118 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.416125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.416136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.416163 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.419134 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.419844 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.419852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.419855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.420006 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.420416 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.420419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.420421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.421035 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.421289 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.421702 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.422086 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":506,"thread_start_us":294,"threads_started":1}
I20260928 05:23:34.423162 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.407921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33568)
0928 05:23:34.408534 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.408573 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.408707 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.409029 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.409036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.409755 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.410058 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.410070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.411085 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.411089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.412448 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.412462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.413265 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.413273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.413449 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.414068 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.414093 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.416361 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.419599 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.419609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.419621 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.420049 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.420383 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.420386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.420389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.420532 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.421166 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.421173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.421386 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.421746 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.422248 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":52,"thread_start_us":303,"threads_started":1}
I20260928 05:23:34.424805 15361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.253@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
I20260928 05:23:34.435563 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.425524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56352)
0928 05:23:34.425733 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.425750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.425832 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.426035 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.426038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.426312 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.426458 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.426465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.427336 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.427339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.428270 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.428280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.428942 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.428946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.429073 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.429500 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.429515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.431268 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.433801 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.433805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.433807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.434130 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.434427 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.434430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.434432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.434545 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.434944 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.434947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.435043 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.435138 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.435314 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 05:23:34.435708 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.425729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.8.124.253:46597)
0928 05:23:34.425848 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.425851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.425867 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:34.425900 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.425903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.425947 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.426014 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.426572 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.427232 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.428421 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.428813 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.428919 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.431434 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.431439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.431442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.431466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.433698 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.434216 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.434220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.434222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.434324 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.434621 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.434624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.434626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.434859 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.434958 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.435198 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.435443 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":40}
I20260928 05:23:34.439908 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Bootstrap starting.
I20260928 05:23:34.442726 15361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:34.443611 15361 log.cc:824] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:34.445995 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: No bootstrap required, opened a new log
I20260928 05:23:34.452433 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } }
I20260928 05:23:34.452697 15361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:34.452764 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6bb68f784046f990e6314c002b855e, State: Initialized, Role: FOLLOWER
I20260928 05:23:34.453224 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } }
I20260928 05:23:34.453971 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } } }
I20260928 05:23:34.454242 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:34.454792 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:23:34.462441 15378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:23:34.462569 15378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:23:34.464609 15294 master_runner.cc:428] Detected that this master 4d6bb68f784046f990e6314c002b855e is joining an existing cluster
I20260928 05:23:34.464704 15294 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.124.253:46597
I20260928 05:23:34.483034 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.468241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.8.124.254:36501)
0928 05:23:34.468403 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.468408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.469110 (+ 702us) server_negotiation.cc:407] Connection header received
0928 05:23:34.469319 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.469324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.469398 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.469465 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.470704 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.471666 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.473660 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.474246 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.474340 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.476666 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.476672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.476677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.476710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.479991 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.480803 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.480809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.480813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.480967 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.481369 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.481374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.481377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.481754 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.481862 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.482442 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.482671 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:23:34.484234 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.468360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33572)
0928 05:23:34.469001 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.469020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.469154 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.469655 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.469659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.470209 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.470578 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.470586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.471787 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.471792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.473485 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.473503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.474552 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.474556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.474726 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.475279 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.475299 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.476478 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.480128 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.480133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.480138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.480684 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.481105 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.481110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.481114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.481263 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.481946 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.481952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.482140 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.483502 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.483806 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":4,"thread_start_us":317,"threads_started":1}
I20260928 05:23:34.499622 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.468645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.8.124.253:46597)
0928 05:23:34.468769 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.468773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.469689 (+ 916us) server_negotiation.cc:407] Connection header received
0928 05:23:34.469825 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.469830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.469900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.469974 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.471260 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.472207 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.473512 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.474180 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.474308 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.493146 (+ 18838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.493153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.493158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.493208 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.496779 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.497567 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.497572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.497576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.497737 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.498133 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.498138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.498141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.498509 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.498618 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.499084 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.499287 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 05:23:34.500881 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.468377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56362)
0928 05:23:34.469556 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.469572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.469705 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.470147 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.470151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.470806 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.471127 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.471139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.472371 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.472377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.473371 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.473387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.474359 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.474366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.474545 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.475467 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.475489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.492962 (+ 17473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.496945 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.496950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.496954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.497454 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.497860 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.497864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.497868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.498023 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.498690 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.498694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.498834 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.500168 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.500438 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":75,"thread_start_us":301,"threads_started":1}
I20260928 05:23:34.509868 15156 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.254@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.124.252@KRBTEST.COM; defaulting to no policy
I20260928 05:23:34.518596 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.489784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.8.124.253:46597)
0928 05:23:34.500136 (+ 10352us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.500142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.503045 (+ 2903us) server_negotiation.cc:407] Connection header received
0928 05:23:34.503649 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.503654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.503749 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.503826 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.505132 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.505889 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.507516 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.507922 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.508004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.512288 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.512294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.512298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.512330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.515743 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.516561 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.516567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.516570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.516724 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.517130 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.517135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.517138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.517461 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.517631 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.518144 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.518305 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10257,"thread_start_us":13392,"threads_started":1}
I20260928 05:23:34.519016 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.489765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56376)
0928 05:23:34.502921 (+ 13156us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.502963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.503443 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.503925 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.503933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.504703 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.505004 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.505017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.506009 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.506015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.507403 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.507417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.508061 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.508066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.508230 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.508847 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.508881 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.512040 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.515872 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.515885 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.515892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.516440 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.516835 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.516840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.516844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.517014 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.517656 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.517665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.517859 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.518062 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.518379 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12912,"thread_start_us":13288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.252@KRBTEST.COM" created.
I20260928 05:23:34.521013 14833 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.124.253:46597: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4d6bb68f784046f990e6314c002b855e"
member_type: NON_VOTER
last_known_addr {
host: "127.8.124.253"
port: 46597
}
attrs {
promote: true
}
}
dest_uuid: "81e8f1656733456a9c4d3d123be61db1"
cas_config_opid_index: -1
I20260928 05:23:34.522166 14833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } }
I20260928 05:23:34.542675 15391 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4d6bb68f784046f990e6314c002b855e (127.8.124.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } }
I20260928 05:23:34.543852 15367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:34.544186 15367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } }
I20260928 05:23:34.545032 15336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:34.545849 15367 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501)
I20260928 05:23:34.546587 15391 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.124.253:46597
I20260928 05:23:34.546492 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d6bb68f784046f990e6314c002b855e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e8f1656733456a9c4d3d123be61db1" is_pre_election: true
I20260928 05:23:34.547353 15390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } }
I20260928 05:23:34.547668 15390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:34.548027 15313 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4d6bb68f784046f990e6314c002b855e; no voters: 81e8f1656733456a9c4d3d123be61db1
I20260928 05:23:34.549078 15336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 05:23:34.549294 15367 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:23:34.550029 15294 master.cc:537] Master@127.8.124.253:46597 shutting down...
I20260928 05:23:34.550344 15391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:23:34.553817 14818 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 05:23:34.555351 14818 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 -> Peer 4d6bb68f784046f990e6314c002b855e (127.8.124.253:46597): Couldn't send request to peer 4d6bb68f784046f990e6314c002b855e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.252.keytab.
Entry for principal oryx/127.8.124.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.252.keytab.
I20260928 05:23:34.558437 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:38333
--webserver_interface=127.8.124.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.252.keytab
--principal=oryx/127.8.124.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:23:34.560735 15294 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:34.561089 15294 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:34.561252 15294 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: stopping tablet replica
I20260928 05:23:34.567638 15294 master.cc:552] Master@127.8.124.253:46597 shutdown complete.
I20260928 05:23:34.567715 15294 master_runner.cc:305] Clearing existing system tablet
I20260928 05:23:34.568122 15294 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:34.568214 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:34.573838 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:23:34.575126 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:34.575405 15294 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:34.575651 15294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 05:23:34.575733 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "4d6bb68f784046f990e6314c002b855e"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "53fac1cfcca3b0128086c105372170c4"
server_key_iv: "e9d3d0f55026e32cdd44e7582d850054"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.579007 15294 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:34.583292 15294 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:23:34.583436 15294 log.cc:1196] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 05:23:34.583763 15294 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Deleting consensus metadata
I20260928 05:23:34.583981 15294 master_runner.cc:315] Copying system tablet from 127.8.124.254:36501
I20260928 05:23:34.586903 15294 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: tablet copy: Beginning tablet copy session from remote peer at address 127.8.124.254:36501
I20260928 05:23:34.599263 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.587550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.8.124.254:36501)
0928 05:23:34.587735 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.587742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.589473 (+ 1731us) server_negotiation.cc:407] Connection header received
0928 05:23:34.589669 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.589675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.589749 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.589831 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:34.590610 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.591302 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.592454 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.592861 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.592969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.594653 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:34.594658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:34.594661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:34.594689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:34.596999 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.597546 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.597551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.597555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.597695 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:34.598030 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:34.598035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:34.598038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:34.598288 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:34.598428 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.598764 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.598957 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:34.599431 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.587498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33582)
0928 05:23:34.589369 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.589387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.589517 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.589870 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.589884 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.590263 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:34.590472 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.590481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.591433 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.591439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.592326 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.592337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.592993 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.592999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.593186 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.593750 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:34.593767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:34.594487 (+ 720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:34.597114 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.597120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.597124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.597447 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.597789 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:34.597794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:34.597798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:34.597939 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:34.598394 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:34.598398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:34.598546 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.598662 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.598944 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":1665,"threads_started":1}
I20260928 05:23:34.600301 14853 tablet_copy_service.cc:140] P 81e8f1656733456a9c4d3d123be61db1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4d6bb68f784046f990e6314c002b855e ({username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:33582)
I20260928 05:23:34.600510 14853 tablet_copy_service.cc:161] P 81e8f1656733456a9c4d3d123be61db1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4d6bb68f784046f990e6314c002b855e at {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:33582: session id = 4d6bb68f784046f990e6314c002b855e-00000000000000000000000000000000
I20260928 05:23:34.603396 14853 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:23:34.605103 15294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:23:34.609346 15294 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: tablet copy: Starting download of 0 data blocks...
I20260928 05:23:34.609615 15294 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: tablet copy: Starting download of 1 WAL segments...
I20260928 05:23:34.611634 15294 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:23:34.614753 14853 tablet_copy_service.cc:342] P 81e8f1656733456a9c4d3d123be61db1: Request end of tablet copy session 4d6bb68f784046f990e6314c002b855e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:33582
I20260928 05:23:34.614900 14853 tablet_copy_service.cc:433] P 81e8f1656733456a9c4d3d123be61db1: ending tablet copy session 4d6bb68f784046f990e6314c002b855e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4d6bb68f784046f990e6314c002b855e
I20260928 05:23:34.623358 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:34.625517 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.635516 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.637887 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:34.638489 15294 server_base.cc:1034] running on GCE node
I20260928 05:23:34.638794 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:34.639320 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:34.640566 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790573014640515 us; error 32 us; skew 500 ppm
I20260928 05:23:34.640756 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:34.642503 15294 init.cc:377] Logged in from keytab as oryx/127.8.124.253@KRBTEST.COM (short username oryx)
I20260928 05:23:34.650362 15294 webserver.cc:460] Webserver started at http://127.8.124.253:39555/ using document root <none> and password file <none>
I20260928 05:23:34.650633 15294 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:34.650704 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:34.652868 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 05:23:34.654306 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:34.654654 15294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 05:23:34.654774 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "4d6bb68f784046f990e6314c002b855e"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "53fac1cfcca3b0128086c105372170c4"
server_key_iv: "e9d3d0f55026e32cdd44e7582d850054"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.654912 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:34.665846 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:34.668900 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:34.691344 15294 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.253:46597
I20260928 05:23:34.691344 15465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.253:46597 every 8 connection(s)
I20260928 05:23:34.692256 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
I20260928 05:23:34.697656 15466 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:23:34.701843 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Bootstrap starting.
I20260928 05:23:34.717164 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:23:34.717568 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Bootstrap complete.
I20260928 05:23:34.718359 15466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } }
I20260928 05:23:34.718560 15466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d6bb68f784046f990e6314c002b855e, State: Initialized, Role: LEARNER
I20260928 05:23:34.718775 15466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } }
I20260928 05:23:34.719509 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } }
I20260928 05:23:34.719759 15466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:34.719868 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: LEARNER
I20260928 05:23:34.726518 15480 catalog_manager.cc:1259] Loaded cluster ID: 5e13c503a25f44569fb8ec805868a783
I20260928 05:23:34.726675 15480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: loading cluster ID for follower catalog manager: success
I20260928 05:23:34.729118 15480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: acquiring CA information for follower catalog manager: success
I20260928 05:23:34.730883 15480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:23:34.928879 15395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:34.929183 15395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:34.929311 15395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:34.936480 15395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:34.936589 15395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:34.936663 15395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:34.936733 15395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:34.936790 15395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:34.946606 15395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.252.keytab
--principal=oryx/127.8.124.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.124.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:34.947857 15395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:34.949102 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:34.961778 15486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.961815 15485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:34.962929 15488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:34.963608 15289 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
I20260928 05:23:34.964047 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.956208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56380)
0928 05:23:34.956458 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:34.956477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:34.956604 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:34.957653 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:34.957658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:34.957686 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:34.957826 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.957834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.959575 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.959579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.961374 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:34.961390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.963288 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.963293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:34.963483 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.963534 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 05:23:34.963626 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:23:34.963769 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 05:23:34.964219 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:34.956355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.8.124.253:46597)
0928 05:23:34.957256 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:34.957262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:34.957282 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:23:34.957360 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:34.957365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:34.957446 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:34.957562 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:34.957952 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.959426 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.961572 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:34.963145 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:34.963265 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:34.963537 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:34.963772 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 05:23:34.963921 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":654,"threads_started":1}
I20260928 05:23:34.965174 15395 server_base.cc:1034] running on GCE node
I20260928 05:23:34.965170 15441 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 05:23:34.965751 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e8f1656733456a9c4d3d123be61db1. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } }
I20260928 05:23:34.965994 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: LEARNER
I20260928 05:23:34.966172 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:34.967561 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:34.968770 15395 hybrid_clock.cc:648] HybridClock initialized: now 1790573014968725 us; error 35 us; skew 500 ppm
I20260928 05:23:34.969072 15395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:34 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:34.972085 15395 init.cc:377] Logged in from keytab as oryx/127.8.124.252@KRBTEST.COM (short username oryx)
I20260928 05:23:34.973500 15395 webserver.cc:460] Webserver started at http://127.8.124.252:46287/ using document root <none> and password file <none>
I20260928 05:23:34.974184 15395 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:34.974278 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:34.974519 15395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:34.977290 15395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/instance:
uuid: "314195dd895c434b93957f375b8d8ae5"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "1d6574ed9b372cdb5accc9e463245c63"
server_key_iv: "dd13bd85ead944fddd1a057c22e9f8eb"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.978024 15395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal/instance:
uuid: "314195dd895c434b93957f375b8d8ae5"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "1d6574ed9b372cdb5accc9e463245c63"
server_key_iv: "dd13bd85ead944fddd1a057c22e9f8eb"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.983028 15395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:23:34.986181 15496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:34.987460 15395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:34.987588 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "314195dd895c434b93957f375b8d8ae5"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "1d6574ed9b372cdb5accc9e463245c63"
server_key_iv: "dd13bd85ead944fddd1a057c22e9f8eb"
server_key_version: "encryptionkey@0"
I20260928 05:23:34.987730 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:35.001969 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:35.005820 15395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:35.006031 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:35.023588 15395 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.252:38333
I20260928 05:23:35.023608 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.252:38333 every 8 connection(s)
I20260928 05:23:35.024868 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
I20260928 05:23:35.025951 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15395
I20260928 05:23:35.028987 15548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:35.035363 15548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.252@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:35.046797 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.027918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.8.124.252:38333)
0928 05:23:35.028651 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.028662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.028703 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:23:35.030339 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.030360 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.030713 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.031036 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.032412 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.033723 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.034862 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.035484 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.035644 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.039885 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.039892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.039901 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.039934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.043068 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.043653 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.043659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.043664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.043814 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.044149 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.044152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.044154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.044677 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.044964 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.045378 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.045758 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260928 05:23:35.051534 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.037659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.8.124.254:36501)
0928 05:23:35.037831 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.037837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.038435 (+ 598us) server_negotiation.cc:407] Connection header received
0928 05:23:35.038592 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.038598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.038670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.038776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.039747 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.040507 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.042418 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.042844 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.042938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.046101 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.046107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.046111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.046149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.049125 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.049783 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.049789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.049793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.049910 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.050229 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.050234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.050236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.050482 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.050600 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.051025 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.051216 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:23:35.051561 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.037397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33590)
0928 05:23:35.038285 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.038326 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.038482 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.038809 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.038813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.039384 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.039605 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.039613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.040629 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.040634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.042295 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.042308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.043014 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.043021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.043181 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.043690 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.043710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.045881 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.049237 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.049246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.049253 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.049677 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.050002 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.050007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.050011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.050129 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.050595 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.050602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.050750 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.050904 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.051168 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":540,"threads_started":1}
I20260928 05:23:35.052802 15548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.252@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
I20260928 05:23:35.063503 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.053418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56386)
0928 05:23:35.053566 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.053580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.053688 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.053922 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.053925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.054174 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.054293 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.054300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.055281 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.055285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.056244 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.056255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.056853 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.056857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.056960 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.057410 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.057425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.059191 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.061759 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.061762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.061764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.062083 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.062389 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.062400 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.062403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.062521 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.062909 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.062911 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.063010 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.063105 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.063283 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:23:35.063526 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.053545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.8.124.253:46597)
0928 05:23:35.053705 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.053709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.053725 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:35.053769 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.053773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.053825 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.053904 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.054399 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.055158 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.056344 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.056740 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.056827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.059338 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.059343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.059347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.059379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.061666 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.062162 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.062166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.062168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.062278 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.062600 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.062603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.062605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.062822 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.062931 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.063165 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.063317 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:23:35.064441 15548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573014, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.252@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
I20260928 05:23:35.074646 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.064898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.0.0.1:48534)
0928 05:23:35.065075 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.065092 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.065173 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.065361 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.065364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.065679 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.065802 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.065809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.066621 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.066625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.067426 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.067436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.068019 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.068023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.068150 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.068611 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.068626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.070442 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.072861 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.072865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.072867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.073174 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.073449 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.073452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.073454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.073577 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.073995 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.073998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.074108 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.074201 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.074359 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 05:23:35.074692 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.065079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.8.124.252:38333)
0928 05:23:35.065182 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.065185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.065198 (+ 13us) server_negotiation.cc:407] Connection header received
0928 05:23:35.065231 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.065234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.065277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.065344 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.065906 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.066524 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.067527 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.067902 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.067983 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.070566 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.070570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.070573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.070598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.072761 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.073255 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.073259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.073261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.073367 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.073675 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.073678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.073680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.073913 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.074008 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.074289 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.074473 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":31}
I20260928 05:23:35.079018 15548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Bootstrap starting.
I20260928 05:23:35.081905 15548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:35.082803 15548 log.cc:824] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:35.085036 15548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: No bootstrap required, opened a new log
I20260928 05:23:35.091711 15548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } }
I20260928 05:23:35.092098 15548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:35.092193 15548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314195dd895c434b93957f375b8d8ae5, State: Initialized, Role: FOLLOWER
I20260928 05:23:35.092664 15548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } }
I20260928 05:23:35.093472 15554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } } }
I20260928 05:23:35.093851 15554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:35.094511 15548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:35.101126 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14894
W20260928 05:23:35.103487 15565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:23:35.103703 15565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:23:35.107117 15395 master_runner.cc:428] Detected that this master 314195dd895c434b93957f375b8d8ae5 is joining an existing cluster
I20260928 05:23:35.107231 15395 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.124.252:38333
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.193@KRBTEST.COM
I20260928 05:23:35.120924 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.102139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.8.124.193:33569)
0928 05:23:35.102697 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.102705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.102731 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:23:35.102793 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.102800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.103055 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.103914 (+ 859us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.104804 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.106443 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.107891 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.108600 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.108761 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.112869 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.112879 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.112891 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.112927 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.116639 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.117495 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.117505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.117509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.117684 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.118146 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.118151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.118155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.118579 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.118884 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.120182 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.120463 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":248,"threads_started":1}
I20260928 05:23:35.122512 15017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:35.131036 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.113872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.8.124.253:46597)
0928 05:23:35.114085 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.114090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.114722 (+ 632us) server_negotiation.cc:407] Connection header received
0928 05:23:35.114975 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.114980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.115054 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.115120 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.116528 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.117628 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.119646 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.120283 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.120388 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.124161 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.124168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.124174 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.124216 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.127659 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.128514 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.128520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.128524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.128684 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.129210 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.129216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.129220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.129670 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.129792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.130425 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.130664 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:23:35.132089 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.113993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56392)
0928 05:23:35.114612 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.114631 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.114825 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.115318 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.115323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.116005 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.116353 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.116365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.117799 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.117803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.119471 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.119486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.120919 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.120929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.121110 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.122748 (+ 1638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.122770 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.123947 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.127878 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.127883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.127886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.128369 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.128821 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.128826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.128829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.129079 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.129883 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.129892 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.130115 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.131259 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.131591 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":647,"threads_started":1}
I20260928 05:23:35.143299 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.115353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.8.124.252:38333)
0928 05:23:35.115528 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.115533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.126038 (+ 10505us) server_negotiation.cc:407] Connection header received
0928 05:23:35.126379 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.126384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.126448 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.126506 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.127721 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.128829 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.130760 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.131434 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.131527 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.136444 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.136450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.136455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.136492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.140342 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.141100 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.141105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.141109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.141246 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.141638 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.141642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.141645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.142000 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.142133 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.142750 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.142959 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:23:35.144414 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.115072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.0.0.1:48536)
0928 05:23:35.125951 (+ 10879us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.125971 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.126238 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.126671 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.126675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.127284 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.127586 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.127597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.129033 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.129038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.130624 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.130638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.131814 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.131821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.131992 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.132585 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.132606 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.134882 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.140480 (+ 5598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.140485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.140489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.140988 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.141378 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.141382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.141386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.141525 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.142338 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.142343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.142502 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.143713 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.143992 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10723,"thread_start_us":10966,"threads_started":1}
I20260928 05:23:35.147394 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.120990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33598)
0928 05:23:35.133411 (+ 12421us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.133430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.133632 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.134123 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.134128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.134667 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.134879 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.134890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.136352 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.136358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.138322 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.138335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.139625 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.139631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.139804 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.140340 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.140360 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.141470 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.144842 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.144847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.144850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.145270 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.145629 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.145634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.145638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.145772 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.146452 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.146457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.146589 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.146733 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.147019 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12301,"thread_start_us":16249,"threads_started":1}
I20260928 05:23:35.149624 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.114468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.8.124.254:36501)
0928 05:23:35.114636 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.114642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.133513 (+ 18871us) server_negotiation.cc:407] Connection header received
0928 05:23:35.133772 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.133778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.133861 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.133930 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.135059 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.136168 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.138505 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.139316 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.139444 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.141674 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.141681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.141685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.141720 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.144742 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.145370 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.145382 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.145386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.145505 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.145910 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.145915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.145918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.146262 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.146396 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.147928 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.148150 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.254@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
I20260928 05:23:35.169509 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.155034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.0.0.1:48546)
0928 05:23:35.155175 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.155194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.155293 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.155673 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.155677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.156118 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.156314 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.156326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.157754 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.157760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.159130 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.159144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.160084 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.160090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.160254 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.160818 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.160840 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.163363 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.167054 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.167059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.167063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.167531 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.167896 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.167901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.167904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.168049 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.168588 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.168593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.168743 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.168897 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.169156 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 05:23:35.169520 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.155064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.8.124.252:38333)
0928 05:23:35.155211 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.155216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.155238 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:23:35.155441 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.155447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.155530 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.155624 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.156482 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.157636 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.159283 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.159933 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.160051 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.163544 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.163551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.163556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.163590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.166928 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.167656 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.167661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.167665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.167795 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.168156 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.168161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.168165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.168479 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.168607 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.168986 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.169188 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:35.171211 14833 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.124.252:38333: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "314195dd895c434b93957f375b8d8ae5"
member_type: NON_VOTER
last_known_addr {
host: "127.8.124.252"
port: 38333
}
attrs {
promote: true
}
}
dest_uuid: "81e8f1656733456a9c4d3d123be61db1"
cas_config_opid_index: 5
I20260928 05:23:35.172183 14833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } }
I20260928 05:23:35.173898 15390 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 314195dd895c434b93957f375b8d8ae5 (127.8.124.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.174580 15523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:35.177703 15390 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.124.252:38333
I20260928 05:23:35.178457 15441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 05:23:35.185199 15395 master.cc:537] Master@127.8.124.252:38333 shutting down...
I20260928 05:23:35.186509 15523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 05:23:35.191396 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.191634 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:35.190915 15576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260928 05:23:35.192528 15390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:35.197214 15441 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 314195dd895c434b93957f375b8d8ae5 (127.8.124.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.201386 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: NON_VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: true } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.201781 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: LEARNER
I20260928 05:23:35.201994 15577 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: attempting to promote NON_VOTER 4d6bb68f784046f990e6314c002b855e to VOTER
I20260928 05:23:35.202601 15577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } }
I20260928 05:23:35.204454 15441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 LEARNER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 05:23:35.204908 15390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 05:23:35.206769 15576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4d6bb68f784046f990e6314c002b855e (127.8.124.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.209748 15441 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4d6bb68f784046f990e6314c002b855e (127.8.124.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.212268 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.212447 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:35.212858 15390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.213042 15390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:35.213392 15576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.213580 15576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:35.220491 15395 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:35.220892 15395 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:35.221102 15395 tablet_replica.cc:330] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: stopping tablet replica
W20260928 05:23:35.221215 14818 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 -> Peer 314195dd895c434b93957f375b8d8ae5 (127.8.124.252:38333): Couldn't send request to peer 314195dd895c434b93957f375b8d8ae5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 05:23:35.227828 15395 master.cc:552] Master@127.8.124.252:38333 shutdown complete.
I20260928 05:23:35.227921 15395 master_runner.cc:305] Clearing existing system tablet
I20260928 05:23:35.228291 15395 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:35.228391 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:35.235008 15395 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.002s
I20260928 05:23:35.238744 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:35.239055 15395 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:35.239303 15395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 05:23:35.239396 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "314195dd895c434b93957f375b8d8ae5"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "1d6574ed9b372cdb5accc9e463245c63"
server_key_iv: "dd13bd85ead944fddd1a057c22e9f8eb"
server_key_version: "encryptionkey@0"
I20260928 05:23:35.244248 15395 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:35.250842 15395 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:23:35.250985 15395 log.cc:1196] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 05:23:35.251428 15395 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Deleting consensus metadata
I20260928 05:23:35.251665 15395 master_runner.cc:315] Copying system tablet from 127.8.124.254:36501
I20260928 05:23:35.254037 15395 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.8.124.254:36501
I20260928 05:23:35.266119 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14894
I20260928 05:23:35.272140 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.254809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.8.124.254:36501)
0928 05:23:35.255036 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.255042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.255525 (+ 483us) server_negotiation.cc:407] Connection header received
0928 05:23:35.255697 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.255703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.255783 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.255887 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.256812 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.257932 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.259746 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.260314 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.260432 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.262764 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.262770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.262776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.262815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.269737 (+ 6922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.270333 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.270339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.270343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.270513 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.270853 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.270857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.270860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.271103 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.271241 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.271604 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.271800 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 05:23:35.272248 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.254772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:33612)
0928 05:23:35.255404 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.255430 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.255579 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.255919 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.255923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.256399 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.256647 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.256659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.258090 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.258096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.259602 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.259617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.260641 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.260646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.260798 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.261466 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.261487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.262574 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.269852 (+ 7278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.269857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.269862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.270218 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.270618 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.270623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.270627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.270758 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.271211 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.271216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.271377 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.271496 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.271800 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":357,"threads_started":1}
I20260928 05:23:35.273028 14853 tablet_copy_service.cc:140] P 81e8f1656733456a9c4d3d123be61db1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 314195dd895c434b93957f375b8d8ae5 ({username='oryx', principal='oryx/127.8.124.252@KRBTEST.COM'} at 127.0.0.1:33612)
I20260928 05:23:35.273168 14853 tablet_copy_service.cc:161] P 81e8f1656733456a9c4d3d123be61db1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 314195dd895c434b93957f375b8d8ae5 at {username='oryx', principal='oryx/127.8.124.252@KRBTEST.COM'} at 127.0.0.1:33612: session id = 314195dd895c434b93957f375b8d8ae5-00000000000000000000000000000000
I20260928 05:23:35.275382 14853 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:23:35.276808 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:23:35.281167 15395 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: tablet copy: Starting download of 0 data blocks...
I20260928 05:23:35.281399 15395 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: tablet copy: Starting download of 1 WAL segments...
I20260928 05:23:35.283267 15395 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:23:35.286103 14853 tablet_copy_service.cc:342] P 81e8f1656733456a9c4d3d123be61db1: Request end of tablet copy session 314195dd895c434b93957f375b8d8ae5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.124.252@KRBTEST.COM'} at 127.0.0.1:33612
I20260928 05:23:35.286203 14853 tablet_copy_service.cc:433] P 81e8f1656733456a9c4d3d123be61db1: ending tablet copy session 314195dd895c434b93957f375b8d8ae5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 314195dd895c434b93957f375b8d8ae5
I20260928 05:23:35.287141 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:33569
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:36501
--webserver_port=44847
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:23:35.297782 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:35.300137 15591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:35.309900 15592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:35.314232 15594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:35.315212 15395 server_base.cc:1034] running on GCE node
I20260928 05:23:35.315588 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:35.316191 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:35.317392 15395 hybrid_clock.cc:648] HybridClock initialized: now 1790573015317351 us; error 21 us; skew 500 ppm
I20260928 05:23:35.317639 15395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573015, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:35.319679 15395 init.cc:377] Logged in from keytab as oryx/127.8.124.252@KRBTEST.COM (short username oryx)
I20260928 05:23:35.322451 15395 webserver.cc:460] Webserver started at http://127.8.124.252:40043/ using document root <none> and password file <none>
I20260928 05:23:35.322749 15395 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:35.322846 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:35.325665 15395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 05:23:35.327498 15601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:35.327965 15395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 05:23:35.328082 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "314195dd895c434b93957f375b8d8ae5"
format_stamp: "Formatted at 2026-09-28 05:23:34 on dist-test-slave-515x"
server_key: "1d6574ed9b372cdb5accc9e463245c63"
server_key_iv: "dd13bd85ead944fddd1a057c22e9f8eb"
server_key_version: "encryptionkey@0"
I20260928 05:23:35.328243 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:35.348958 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:35.353012 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:35.377542 15395 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.252:38333
I20260928 05:23:35.377591 15652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.252:38333 every 8 connection(s)
I20260928 05:23:35.378533 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
I20260928 05:23:35.383657 15653 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:23:35.387488 15653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Bootstrap starting.
I20260928 05:23:35.404929 15653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:23:35.405463 15653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: Bootstrap complete.
I20260928 05:23:35.406523 15653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } }
I20260928 05:23:35.406805 15653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 314195dd895c434b93957f375b8d8ae5, State: Initialized, Role: LEARNER
I20260928 05:23:35.407110 15653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } }
I20260928 05:23:35.407984 15656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.408300 15653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:35.408510 15656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:23:35.447212 15667 catalog_manager.cc:1259] Loaded cluster ID: 5e13c503a25f44569fb8ec805868a783
I20260928 05:23:35.447328 15667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: loading cluster ID for follower catalog manager: success
I20260928 05:23:35.450640 15667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: acquiring CA information for follower catalog manager: success
I20260928 05:23:35.452358 15667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:23:35.662091 15590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:35.662330 15590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:35.662426 15590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:35.669396 15590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:35.669505 15590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:35.669698 15590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:35.679171 15590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:33569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=44847
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:35.680464 15590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:35.681730 15590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:35.694245 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:35.694717 15673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:35.694692 15672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:35.695171 15590 server_base.cc:1034] running on GCE node
I20260928 05:23:35.695717 15590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:35.696606 15590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:35.697798 15590 hybrid_clock.cc:648] HybridClock initialized: now 1790573015697758 us; error 34 us; skew 500 ppm
I20260928 05:23:35.698076 15590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573015, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:35.700865 15590 init.cc:377] Logged in from keytab as oryx/127.8.124.193@KRBTEST.COM (short username oryx)
I20260928 05:23:35.702284 15590 webserver.cc:460] Webserver started at http://127.8.124.193:44847/ using document root <none> and password file <none>
I20260928 05:23:35.702899 15590 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:35.703008 15590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:35.707914 15590 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:23:35.710558 15682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:35.711951 15590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:35.712090 15590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9"
format_stamp: "Formatted at 2026-09-28 05:23:32 on dist-test-slave-515x"
server_key: "3bfe8e2ab3862e2796cca00e82c5c012"
server_key_iv: "b9c10bdfb1c7e870847f0589f8bc57d5"
server_key_version: "encryptionkey@0"
I20260928 05:23:35.713070 15590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:35.744926 15590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:35.748974 15590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:35.749219 15590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:35.750149 15590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:35.751430 15590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:35.751513 15590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:35.751612 15590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:35.751680 15590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:35.780404 15590 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33569
I20260928 05:23:35.780453 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33569 every 8 connection(s)
I20260928 05:23:35.782066 15590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:35.782822 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15590
I20260928 05:23:35.783133 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15028
I20260928 05:23:35.785097 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.777026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.0.0.1:48556)
0928 05:23:35.778601 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.778622 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.778766 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.779201 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.779207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.779241 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:35.779444 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.779455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.780904 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.780910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.782672 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.782688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.784289 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.784296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.784495 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.784536 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.784604 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.784756 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"thread_start_us":1373,"threads_started":1}
I20260928 05:23:35.794140 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.777119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.8.124.252:38333)
0928 05:23:35.778897 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.778901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.778918 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:35.778978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.778981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.779046 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.779142 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:35.779592 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.780758 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.782885 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.784079 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.785477 (+ 1398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.785732 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.785789 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.785948 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":1553,"threads_started":1}
I20260928 05:23:35.795063 15628 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 05:23:35.795888 15656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e8f1656733456a9c4d3d123be61db1. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:35.796162 15656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: LEARNER
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.194@KRBTEST.COM
I20260928 05:23:35.812556 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.783931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.8.124.194:37565)
0928 05:23:35.790171 (+ 6240us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.790177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.790201 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:35.790251 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.790256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.790470 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.790604 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.792650 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.794366 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.796396 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.797286 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.803807 (+ 6521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.804114 (+ 307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.804126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.804147 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.804178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.808101 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.809118 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.809128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.809133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.809298 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.809777 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.809782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.809786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.810218 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.811671 (+ 1453us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.811840 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.812109 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6093,"thread_start_us":5945,"threads_started":1}
I20260928 05:23:35.814043 15151 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573015, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573015, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
Sep 28 05:23:35 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573015, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
I20260928 05:23:35.835999 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.787576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:58129 (local address 127.8.124.254:36501)
0928 05:23:35.813302 (+ 25726us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.813308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.813869 (+ 561us) server_negotiation.cc:407] Connection header received
0928 05:23:35.816358 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.816364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.816465 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.816565 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.819077 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.820354 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.823327 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.823921 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.824035 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.828535 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.828542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.828547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.828606 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.831815 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.833038 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.833043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.833047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.833188 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.833687 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.833693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.833696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.834096 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.834222 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.835310 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.835556 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25517,"thread_start_us":25609,"threads_started":1}
I20260928 05:23:35.838208 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.787159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:58129)
0928 05:23:35.813707 (+ 26548us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.813723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.815889 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.816876 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.816887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.817840 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.818804 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.818827 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.820581 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.820587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.823126 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.823140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.824198 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.824205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.824526 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.825230 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.825258 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.828287 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.831943 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.831953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.831971 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.832915 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.833314 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.833319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.833323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.833541 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.834313 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.834321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.834849 (+ 528us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.836241 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.836816 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26415,"mutex_wait_us":2,"spinlock_wait_cycles":22400,"thread_start_us":26712,"threads_started":1}
I20260928 05:23:35.839030 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.815554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.8.124.193:60421)
0928 05:23:35.824154 (+ 8600us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.824177 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.824280 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.824782 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.824787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.825321 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.825650 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.825661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.827088 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.827093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.828630 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.828642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.829619 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.829626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.829815 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.830350 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.830372 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.832840 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.836397 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.836402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.836406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.836979 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.837357 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.837364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.837509 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.838175 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.838179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.838313 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.838428 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.838671 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8478,"thread_start_us":8330,"threads_started":1}
I20260928 05:23:35.839448 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.786160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:52437 (local address 127.8.124.252:38333)
0928 05:23:35.796735 (+ 10575us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.796740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.811262 (+ 14522us) server_negotiation.cc:407] Connection header received
0928 05:23:35.816230 (+ 4968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.816237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.816325 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.816395 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.824629 (+ 8234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.825668 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.827549 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.828139 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.828229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.832777 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.832783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.832787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.832825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.836253 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837225 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.837230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.837233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.837375 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837792 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.837796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.837799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.838100 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.838206 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.838921 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.839124 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10484,"thread_start_us":24193,"threads_started":1}
I20260928 05:23:35.839730 15798 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:35.840000 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.814729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:60421 (local address 127.8.124.253:46597)
0928 05:23:35.823854 (+ 9125us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:35.823860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:35.824245 (+ 385us) server_negotiation.cc:407] Connection header received
0928 05:23:35.824424 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:35.824429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:35.824515 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:35.824604 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:35.825840 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.826939 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.828777 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.829404 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.829524 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.833053 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:35.833059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:35.833063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:35.833099 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:35.836256 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837107 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.837112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.837115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.837241 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837651 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:35.837655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:35.837658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:35.837977 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:35.838115 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:35.839464 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0928 05:23:35.839674 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9004,"thread_start_us":8881,"threads_started":1}
I20260928 05:23:35.840027 15798 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:35.840718 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:35.785496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.8.124.193:52437)
0928 05:23:35.811087 (+ 25591us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:35.811155 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:35.815886 (+ 4731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:35.816876 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:35.816888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:35.817906 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:35.824475 (+ 6569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.824486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.825837 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.825841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.827391 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:35.827404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:35.828513 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:35.828519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:35.828676 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:35.829239 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:35.829260 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:35.832588 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:35.836685 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.836690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.836693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.837113 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.837499 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:35.837503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:35.837505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:35.837680 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:35.838514 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:35.838521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:35.838672 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 05:23:35.839937 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0928 05:23:35.840411 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25208,"mutex_wait_us":5929,"thread_start_us":27318,"threads_started":1}
I20260928 05:23:35.840888 15798 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:35.841676 15797 heartbeater.cc:347] Connected to a master server at 127.8.124.252:38333
I20260928 05:23:35.841804 15797 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:35.842077 15803 heartbeater.cc:347] Connected to a master server at 127.8.124.253:46597
I20260928 05:23:35.842211 15803 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:35.842267 15797 heartbeater.cc:511] Master 127.8.124.252:38333 requested a full tablet report, sending...
I20260928 05:23:35.842604 14833 ts_manager.cc:194] Re-registered known tserver with Master: a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569)
I20260928 05:23:35.842622 15803 heartbeater.cc:511] Master 127.8.124.253:46597 requested a full tablet report, sending...
I20260928 05:23:35.843662 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:58129
I20260928 05:23:35.843954 15618 ts_manager.cc:194] Registered new tserver with Master: a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569)
I20260928 05:23:35.844934 15431 ts_manager.cc:194] Registered new tserver with Master: a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569)
I20260928 05:23:35.910027 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15028
I20260928 05:23:35.923589 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:37565
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:36501
--webserver_port=35219
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:36.229477 15810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:36.229768 15810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:36.229867 15810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:36.236795 15810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:36.236889 15810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:36.237001 15810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:36.245958 15810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:37565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=35219
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:36.247092 15810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:36.248234 15810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:36.260057 15816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:36.260015 15815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:36.259989 15818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:36.260632 15810 server_base.cc:1034] running on GCE node
I20260928 05:23:36.261198 15810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:36.262082 15810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:36.263264 15810 hybrid_clock.cc:648] HybridClock initialized: now 1790573016263222 us; error 37 us; skew 500 ppm
I20260928 05:23:36.263554 15810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:36.266602 15810 init.cc:377] Logged in from keytab as oryx/127.8.124.194@KRBTEST.COM (short username oryx)
I20260928 05:23:36.267841 15810 webserver.cc:460] Webserver started at http://127.8.124.194:35219/ using document root <none> and password file <none>
I20260928 05:23:36.268469 15810 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:36.268586 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:36.273543 15810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:36.275982 15825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.277150 15810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:36.277287 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "23f5377cb1844d3980f2fa588288120a"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "31d26d7024830ea13122f21cb6742289"
server_key_iv: "fcbb8e20fa27dab57aeb4a4b56996794"
server_key_version: "encryptionkey@0"
I20260928 05:23:36.278236 15810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:36.307016 15810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:36.311200 15810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:36.311440 15810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:36.312318 15810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:36.313637 15810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:36.313728 15810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.313827 15810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:36.313894 15810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.330812 15576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: NON_VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: true } } }
I20260928 05:23:36.331140 15576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:36.340333 15896 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: attempting to promote NON_VOTER 314195dd895c434b93957f375b8d8ae5 to VOTER
I20260928 05:23:36.341135 15896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } }
I20260928 05:23:36.343483 15628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 LEARNER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:23:36.343595 15441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Refusing update from remote peer 81e8f1656733456a9c4d3d123be61db1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:23:36.344081 15576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:23:36.344480 15896 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:23:36.348327 15896 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 314195dd895c434b93957f375b8d8ae5 (127.8.124.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.348984 15628 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 314195dd895c434b93957f375b8d8ae5 (127.8.124.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.350845 15441 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 314195dd895c434b93957f375b8d8ae5 (127.8.124.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.352028 15896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.352298 15896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:36.352605 15896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.352847 15896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:36.353142 15896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.353379 15896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:36.355813 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.356784 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:36.356815 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81e8f1656733456a9c4d3d123be61db1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:36.357249 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:36.360206 15810 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:37565
I20260928 05:23:36.360261 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:37565 every 8 connection(s)
I20260928 05:23:36.361490 15810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:36.362336 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15810
I20260928 05:23:36.362612 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15161
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.195@KRBTEST.COM
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
I20260928 05:23:36.403827 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.363497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.8.124.195:38645)
0928 05:23:36.378280 (+ 14783us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.378288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.381792 (+ 3504us) server_negotiation.cc:407] Connection header received
0928 05:23:36.381946 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.381955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.382223 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.382428 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.384804 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.386420 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.388365 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.389242 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.389366 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.396067 (+ 6701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.396081 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.396099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.396139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.399954 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.400776 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.400786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.400789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.400953 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.401350 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.401355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.401359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.401798 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.402097 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.403275 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.403518 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14620,"thread_start_us":16553,"threads_started":1}
I20260928 05:23:36.405344 15284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:36.413201 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.370170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:46153 (local address 127.8.124.252:38333)
0928 05:23:36.376356 (+ 6186us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.376363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.378935 (+ 2572us) server_negotiation.cc:407] Connection header received
0928 05:23:36.378998 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.379004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.379100 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.379204 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.384542 (+ 5338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.385703 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.395324 (+ 9621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.396044 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.396144 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.405356 (+ 9212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.405365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.405371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.405416 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.409092 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.410110 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.410115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.410119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.410254 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.410737 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.410742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.410746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.411061 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.411189 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.412315 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.412671 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6051,"thread_start_us":5895,"threads_started":1}
I20260928 05:23:36.415095 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.365420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:55865 (local address 127.8.124.254:36501)
0928 05:23:36.379723 (+ 14303us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.379729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.379747 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:36.379796 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.379801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.379866 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.379958 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.383252 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.384370 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.389656 (+ 5286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.390281 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.390410 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.405400 (+ 14990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.405408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.405412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.405453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.412289 (+ 6836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.413109 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.413115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.413119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.413267 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.413768 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.413773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.413777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.414130 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.414294 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.414689 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.414847 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14184,"thread_start_us":14057,"threads_started":1}
I20260928 05:23:36.415675 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.369706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.8.124.194:46153)
0928 05:23:36.378699 (+ 8993us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.378718 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.378838 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.380534 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.380547 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.381411 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.382229 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.382253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.385899 (+ 3646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.385904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.387915 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.387930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.396488 (+ 8558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.396495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.396669 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.399972 (+ 3303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.399996 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.405067 (+ 5071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.409303 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.409312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.409327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.409964 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.410389 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.410393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.410397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.410613 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.411253 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.411261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.411708 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.412049 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.414276 (+ 2227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8868,"mutex_wait_us":3,"thread_start_us":2761,"threads_started":1}
I20260928 05:23:36.415745 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.364867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:55865)
0928 05:23:36.372940 (+ 8073us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.373003 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.375619 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.382437 (+ 6818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.382442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.382834 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.383087 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.383097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.388040 (+ 4943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.388045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.389461 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.389473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.394358 (+ 4885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.394365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.394687 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.397422 (+ 2735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.397455 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.405159 (+ 7704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.412459 (+ 7300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.412465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.412469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.412937 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.413448 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.413452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.413456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.413662 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.414283 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.414288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.414471 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.414613 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.414880 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7712,"thread_start_us":13433,"threads_started":1}
I20260928 05:23:36.417474 15947 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:36.417520 15946 heartbeater.cc:347] Connected to a master server at 127.8.124.252:38333
I20260928 05:23:36.417805 15946 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.417811 15947 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.418594 15946 heartbeater.cc:511] Master 127.8.124.252:38333 requested a full tablet report, sending...
I20260928 05:23:36.418619 15947 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:36.419978 14833 ts_manager.cc:194] Re-registered known tserver with Master: 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
I20260928 05:23:36.420075 15618 ts_manager.cc:194] Registered new tserver with Master: 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
I20260928 05:23:36.420954 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:55865
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
I20260928 05:23:36.427172 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.391599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:48823 (local address 127.8.124.253:46597)
0928 05:23:36.402382 (+ 10783us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.402389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.403743 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 05:23:36.403890 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.403895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.403978 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.404082 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.406866 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.408143 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.410193 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.410796 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.410912 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.422328 (+ 11416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.422335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.422340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.422381 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.424960 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.425657 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.425660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.425662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.425767 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.426101 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.426104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.426106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.426343 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.426453 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.426765 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.426922 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10634,"thread_start_us":10462,"threads_started":1}
I20260928 05:23:36.427367 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.391266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.8.124.194:48823)
0928 05:23:36.403630 (+ 12364us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.403650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.403783 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.404254 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.404258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.404932 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.406678 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.406690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.408755 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.408758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.410055 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.410067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.411149 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.411155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.411322 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.417339 (+ 6017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.417359 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.422115 (+ 4756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.425089 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.425094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.425098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.425532 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.425853 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.425857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.425861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.426018 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.426438 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.426443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.426573 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.426694 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.426989 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12224,"mutex_wait_us":5188,"thread_start_us":12045,"threads_started":1}
I20260928 05:23:36.428042 15948 heartbeater.cc:347] Connected to a master server at 127.8.124.253:46597
I20260928 05:23:36.428150 15948 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.428479 15948 heartbeater.cc:511] Master 127.8.124.253:46597 requested a full tablet report, sending...
I20260928 05:23:36.429508 15431 ts_manager.cc:194] Registered new tserver with Master: 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
I20260928 05:23:36.495630 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15161
I20260928 05:23:36.507797 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:38645
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:36501
--webserver_port=36861
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:36.810775 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:36.811025 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:36.811115 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:36.818198 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:36.818291 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:36.818404 15959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:36.827751 15959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:38645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=36861
--tserver_master_addrs=127.8.124.254:36501,127.8.124.253:46597,127.8.124.252:38333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:36.828888 15959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:36.830199 15959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:36.842232 15967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:36.842283 15964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:36.842309 15965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:36.843272 15959 server_base.cc:1034] running on GCE node
I20260928 05:23:36.843823 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:36.844668 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:36.845897 15959 hybrid_clock.cc:648] HybridClock initialized: now 1790573016845830 us; error 46 us; skew 500 ppm
I20260928 05:23:36.846196 15959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:23:36.847153 15798 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:36.849341 15959 init.cc:377] Logged in from keytab as oryx/127.8.124.195@KRBTEST.COM (short username oryx)
I20260928 05:23:36.850705 15959 webserver.cc:460] Webserver started at http://127.8.124.195:36861/ using document root <none> and password file <none>
I20260928 05:23:36.851320 15959 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:36.851425 15959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:36.856595 15959 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:23:36.859174 15974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.860419 15959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:23:36.860549 15959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
format_stamp: "Formatted at 2026-09-28 05:23:33 on dist-test-slave-515x"
server_key: "2dcf3e16be9fff4d3da11b920d15f31f"
server_key_iv: "2c4fcec68159c67be3bbeb5bab04724a"
server_key_version: "encryptionkey@0"
I20260928 05:23:36.861425 15959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:36.898207 15959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:36.902029 15959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:36.902264 15959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:36.903167 15959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:36.904544 15959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:36.904626 15959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.904699 15959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:36.904749 15959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:36.933777 15959 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:38645
I20260928 05:23:36.933863 16086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:38645 every 8 connection(s)
I20260928 05:23:36.935177 15959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:36.945214 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 15959
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.253@KRBTEST.COM
I20260928 05:23:36.965967 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.937999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:42119 (local address 127.8.124.254:36501)
0928 05:23:36.943933 (+ 5934us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.943938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.943954 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:36.944003 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.944007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.944065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.944147 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.948603 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.949733 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.952029 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.952707 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.952822 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.957116 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.957122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.957125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.957156 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.960803 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.962703 (+ 1900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.962710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.962714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.962912 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.963907 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.963912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.963916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.964262 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.964421 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.965296 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.965545 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5833,"thread_start_us":5741,"threads_started":1}
I20260928 05:23:36.967383 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.938765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:42119)
0928 05:23:36.939685 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.939741 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.942351 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.944424 (+ 2073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.944436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.945316 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.946302 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.946327 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.949880 (+ 3553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.949887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.951882 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.951898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.952907 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.952914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.953197 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.954012 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.954045 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.956923 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.962009 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.962018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.962040 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.962584 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.963516 (+ 932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.963521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.963525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.963776 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.964426 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.964435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.964857 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.965184 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.965759 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":43,"spinlock_wait_cycles":16256,"thread_start_us":363,"threads_started":1}
Sep 28 05:23:36 dist-test-slave-515x krb5kdc[14788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573016, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.252@KRBTEST.COM
I20260928 05:23:36.970708 16088 heartbeater.cc:347] Connected to a master server at 127.8.124.254:36501
I20260928 05:23:36.971057 16088 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.972173 16088 heartbeater.cc:511] Master 127.8.124.254:36501 requested a full tablet report, sending...
I20260928 05:23:36.973798 14833 ts_manager.cc:194] Re-registered known tserver with Master: ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:36.974742 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.943259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.8.124.195:43871)
0928 05:23:36.943759 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.943775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.943878 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.954251 (+ 10373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.954257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.954894 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.955251 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.955267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.957919 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.957924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.959362 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.959375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.960413 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.960420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.960604 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.961152 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.961174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.963775 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.967670 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.967674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.967678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.968224 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.969580 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.969586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.969589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.969769 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.973760 (+ 3991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.973765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.973972 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.974091 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.974357 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":75136,"thread_start_us":251,"threads_started":1}
I20260928 05:23:36.974934 14833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:42119
I20260928 05:23:36.976418 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.937997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:33079 (local address 127.8.124.252:38333)
0928 05:23:36.944462 (+ 6465us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.944466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.944479 (+ 13us) server_negotiation.cc:407] Connection header received
0928 05:23:36.944523 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.944526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.944577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.944661 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.946500 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.947652 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.956732 (+ 9080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.957429 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.957556 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.969752 (+ 12196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.969759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.969763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.969799 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.973216 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.974083 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.974089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.974092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.974218 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.974633 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.974638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.974641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.975246 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.975390 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.975863 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.976062 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6368,"thread_start_us":7618,"threads_started":1}
I20260928 05:23:36.977535 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.938765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.8.124.195:33079)
0928 05:23:36.943110 (+ 4345us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:36.943128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:36.943222 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:36.944716 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:36.944721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:36.945431 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:36.946300 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.946325 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.947801 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.947807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.956591 (+ 8784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:36.956603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.957631 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.957639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:36.957795 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.958530 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:36.958550 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:36.969503 (+ 10953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:36.973366 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.973371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.973374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.973960 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.974367 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:36.974371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:36.974375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:36.974500 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:36.975458 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:36.975462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:36.975603 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 05:23:36.976860 (+ 1257us) client_negotiation.cc:240] Negotiation successful
0928 05:23:36.977151 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4188,"mutex_wait_us":348,"thread_start_us":3728,"threads_started":1}
I20260928 05:23:36.978569 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.943350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:43871 (local address 127.8.124.253:46597)
0928 05:23:36.953948 (+ 10598us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.953954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.953973 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:23:36.954029 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.954034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.954106 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.954211 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.956083 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.957784 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.959553 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.960207 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.960881 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.964022 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.964029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.964035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.964118 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.967535 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.969305 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.969310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.969313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.969455 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.973260 (+ 3805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.973265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.973269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.973629 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.975076 (+ 1447us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.975189 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.978209 (+ 3020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10491,"thread_start_us":12756,"threads_started":1}
I20260928 05:23:36.979291 16089 heartbeater.cc:347] Connected to a master server at 127.8.124.253:46597
I20260928 05:23:36.979425 16089 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.979835 16089 heartbeater.cc:511] Master 127.8.124.253:46597 requested a full tablet report, sending...
I20260928 05:23:36.981062 15431 ts_manager.cc:194] Registered new tserver with Master: ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:36.983011 16087 heartbeater.cc:347] Connected to a master server at 127.8.124.252:38333
I20260928 05:23:36.983129 16087 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:36.983430 16087 heartbeater.cc:511] Master 127.8.124.252:38333 requested a full tablet report, sending...
I20260928 05:23:36.984408 15618 ts_manager.cc:194] Registered new tserver with Master: ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:36.985106 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.948875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.8.124.253:46597)
0928 05:23:36.956196 (+ 7321us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.956205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.961788 (+ 5583us) server_negotiation.cc:407] Connection header received
0928 05:23:36.962043 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.962049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.962122 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.962197 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.967896 (+ 5699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.969173 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.971799 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.972468 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.972557 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.978312 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.978318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.978323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.978366 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.981360 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.983064 (+ 1704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.983070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.983073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.983229 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.983653 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.983658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.983661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.983980 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.984207 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.984534 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.984737 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7089,"thread_start_us":4877,"threads_started":1}
I20260928 05:23:36.992882 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.959392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.8.124.254:36501)
0928 05:23:36.964800 (+ 5408us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.964807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.966362 (+ 1555us) server_negotiation.cc:407] Connection header received
0928 05:23:36.966417 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.966422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.966495 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.966596 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.980771 (+ 14175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.981997 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.984647 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.985139 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.985243 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.987966 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.987971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.987974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.988003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.990456 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.991020 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.991026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.991029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.991163 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.991506 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.991511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.991514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.991820 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.992042 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.992308 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.992517 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5287,"thread_start_us":6721,"threads_started":1}
I20260928 05:23:36.995617 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:36.966231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.8.124.252:38333)
0928 05:23:36.976821 (+ 10590us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:36.976826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:36.982007 (+ 5181us) server_negotiation.cc:407] Connection header received
0928 05:23:36.982146 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:36.982151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:36.982235 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:36.982326 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:36.985469 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.986496 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.987823 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:36.988226 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:36.988331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:36.990444 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:36.990450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:36.990455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:36.990486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:36.993233 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.993928 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.993933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.993937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.994052 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:36.994432 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:36.994436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:36.994438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:36.994762 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:36.994948 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:36.995245 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 05:23:36.995401 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10496,"thread_start_us":16026,"threads_started":1}
I20260928 05:23:36.996697 14833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:37.003337 14833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:37.036106 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.021466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.8.124.195:38645)
0928 05:23:37.022351 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.022361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.022397 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:23:37.022526 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.022535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.030068 (+ 7533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.030265 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:37.030721 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.032236 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.033923 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.035166 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.035278 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.035527 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.035657 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.035802 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":559,"threads_started":1}
I20260928 05:23:37.036120 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.021062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38645 (local address 127.0.0.1:46792)
0928 05:23:37.021769 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:37.021790 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:37.021928 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:37.030245 (+ 8317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:37.030249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:37.030270 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:37.030524 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.030532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.032352 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.032358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.033789 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.033800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.035340 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.035345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.035508 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.035539 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:23:37.035622 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:23:37.035746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":362,"threads_started":1}
I20260928 05:23:37.038220 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.021505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33569 (local address 127.0.0.1:58060)
0928 05:23:37.029169 (+ 7664us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:37.029189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:37.029323 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:37.030591 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:37.030596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:37.030623 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:37.031032 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.031042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.034295 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.034300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.035775 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.035786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.037447 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.037453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.037636 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.037666 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:23:37.037742 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 05:23:37.037860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7503,"thread_start_us":7327,"threads_started":1}
I20260928 05:23:37.038517 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.021907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.8.124.193:33569)
0928 05:23:37.026040 (+ 4133us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.026050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.029299 (+ 3249us) server_negotiation.cc:407] Connection header received
0928 05:23:37.029468 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.029476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.029741 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.029930 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:37.031198 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.034197 (+ 2999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.035916 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.037288 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.037420 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.037827 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.037891 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.038232 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3949,"thread_start_us":3845,"threads_started":1}
I20260928 05:23:37.040591 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.022880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:37565 (local address 127.0.0.1:37534)
0928 05:23:37.029170 (+ 6290us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:37.029184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:37.029282 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:37.031902 (+ 2620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:37.031906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:37.031930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:37.032209 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.032217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.033457 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.033461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.037210 (+ 3749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.037224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.039895 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.039901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.040066 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.040098 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:23:37.040174 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 05:23:37.040286 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6199,"thread_start_us":6521,"threads_started":1}
I20260928 05:23:37.040616 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.023066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.8.124.194:37565)
0928 05:23:37.031482 (+ 8416us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.031490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.031513 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:23:37.031580 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.031585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.031744 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.031898 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:37.032352 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.033311 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.038547 (+ 5236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.039711 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.039818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.040013 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.040223 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.040373 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8279,"thread_start_us":8187,"threads_started":1}
I20260928 05:23:37.040350 16022 tablet_service.cc:1467] Processing CreateTablet for tablet 7879172661ba452f95be8d243fac8d8b (DEFAULT_TABLE table=test-table [id=f997dc2683e445b4bb848824aeaaa6fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:37.041786 16022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7879172661ba452f95be8d243fac8d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:37.041786 15730 tablet_service.cc:1467] Processing CreateTablet for tablet 7879172661ba452f95be8d243fac8d8b (DEFAULT_TABLE table=test-table [id=f997dc2683e445b4bb848824aeaaa6fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:37.043062 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7879172661ba452f95be8d243fac8d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:37.043123 15873 tablet_service.cc:1467] Processing CreateTablet for tablet 7879172661ba452f95be8d243fac8d8b (DEFAULT_TABLE table=test-table [id=f997dc2683e445b4bb848824aeaaa6fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:37.044008 15873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7879172661ba452f95be8d243fac8d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:37.054415 16121 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Bootstrap starting.
I20260928 05:23:37.058085 16121 tablet_bootstrap.cc:654] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:37.058248 16123 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Bootstrap starting.
I20260928 05:23:37.059513 16121 log.cc:824] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:37.060839 16122 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Bootstrap starting.
I20260928 05:23:37.061592 16123 tablet_bootstrap.cc:654] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:37.062477 16121 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: No bootstrap required, opened a new log
I20260928 05:23:37.062714 16121 ts_tablet_manager.cc:1397] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:23:37.062786 16123 log.cc:824] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:37.064420 16122 tablet_bootstrap.cc:654] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:37.065582 16122 log.cc:824] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:37.066419 16123 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: No bootstrap required, opened a new log
I20260928 05:23:37.066648 16123 ts_tablet_manager.cc:1397] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:23:37.068573 16122 tablet_bootstrap.cc:492] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: No bootstrap required, opened a new log
I20260928 05:23:37.068846 16122 ts_tablet_manager.cc:1397] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260928 05:23:37.070716 16121 raft_consensus.cc:348] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.071055 16121 raft_consensus.cc:374] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:37.071139 16121 raft_consensus.cc:729] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f5377cb1844d3980f2fa588288120a, State: Initialized, Role: FOLLOWER
I20260928 05:23:37.071674 16121 consensus_queue.cc:260] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.076391 16123 raft_consensus.cc:348] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.076785 16123 raft_consensus.cc:374] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:37.076874 16123 raft_consensus.cc:729] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c01eb21be64ed6b4b9a02bdc0433f9, State: Initialized, Role: FOLLOWER
I20260928 05:23:37.077464 15947 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
I20260928 05:23:37.077448 16123 consensus_queue.cc:260] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.079625 16122 raft_consensus.cc:348] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.080060 16122 raft_consensus.cc:374] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:37.080145 16122 raft_consensus.cc:729] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4a396d62384eb48bfe8bcccf3955ea, State: Initialized, Role: FOLLOWER
I20260928 05:23:37.080801 16122 consensus_queue.cc:260] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.083310 16123 ts_tablet_manager.cc:1428] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260928 05:23:37.084065 16121 ts_tablet_manager.cc:1428] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Time spent starting tablet: real 0.021s user 0.016s sys 0.000s
I20260928 05:23:37.084374 16088 heartbeater.cc:503] Master 127.8.124.254:36501 was elected leader, sending a full tablet report...
I20260928 05:23:37.086354 16122 ts_tablet_manager.cc:1428] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260928 05:23:37.099009 16128 raft_consensus.cc:482] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:37.099289 16128 raft_consensus.cc:504] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.100857 16128 leader_election.cc:288] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645), 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
I20260928 05:23:37.109364 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.101233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:33985 (local address 127.8.124.195:38645)
0928 05:23:37.101423 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.101430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.102011 (+ 581us) server_negotiation.cc:407] Connection header received
0928 05:23:37.102217 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.102222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.102297 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.102390 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:37.103110 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.104182 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.107127 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.108344 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.108508 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.108584 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.108987 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.109112 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:37.109364 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.101293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:60789 (local address 127.8.124.194:37565)
0928 05:23:37.101488 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.101494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.101980 (+ 486us) server_negotiation.cc:407] Connection header received
0928 05:23:37.102217 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.102222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.102297 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.102390 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:37.103110 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.104182 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.107147 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.108334 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.108525 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.108595 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.108987 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.109112 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 05:23:37.109675 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.101106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:37565 (local address 127.8.124.193:60789)
0928 05:23:37.101869 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:37.101888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:37.102053 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:37.102417 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:37.102422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:37.102458 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:37.102946 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.102959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.104322 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.104328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.106947 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.106959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.108583 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.108589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.108857 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.108882 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 05:23:37.108961 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:23:37.109145 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":286,"spinlock_wait_cycles":19072,"thread_start_us":385,"threads_started":1}
I20260928 05:23:37.109800 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.101106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38645 (local address 127.8.124.193:33985)
0928 05:23:37.101919 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:37.101933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:37.102053 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:37.102417 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:37.102422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:37.102454 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:37.102952 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.102960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.104322 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.104328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.106943 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:37.106959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.108569 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.108576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:37.108786 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.108846 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 05:23:37.108961 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:23:37.109145 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":216,"spinlock_wait_cycles":11008,"thread_start_us":470,"threads_started":1}
I20260928 05:23:37.110646 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7879172661ba452f95be8d243fac8d8b" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f5377cb1844d3980f2fa588288120a" is_pre_election: true
I20260928 05:23:37.110644 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7879172661ba452f95be8d243fac8d8b" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" is_pre_election: true
I20260928 05:23:37.111104 15893 raft_consensus.cc:2454] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 0.
I20260928 05:23:37.111104 16042 raft_consensus.cc:2454] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 0.
I20260928 05:23:37.111747 15684 leader_election.cc:302] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c01eb21be64ed6b4b9a02bdc0433f9, ed4a396d62384eb48bfe8bcccf3955ea; no voters:
I20260928 05:23:37.112167 16128 raft_consensus.cc:2781] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:37.112310 16128 raft_consensus.cc:482] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:37.112401 16128 raft_consensus.cc:3037] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:37.114528 16128 raft_consensus.cc:504] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.115079 16128 leader_election.cc:288] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 election: Requested vote from peers ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645), 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565)
I20260928 05:23:37.115461 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7879172661ba452f95be8d243fac8d8b" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
I20260928 05:23:37.115511 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7879172661ba452f95be8d243fac8d8b" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f5377cb1844d3980f2fa588288120a"
I20260928 05:23:37.115625 16042 raft_consensus.cc:3037] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:37.115696 15893 raft_consensus.cc:3037] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:37.117977 16042 raft_consensus.cc:2454] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 1.
I20260928 05:23:37.118058 15893 raft_consensus.cc:2454] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 1.
I20260928 05:23:37.118348 15684 leader_election.cc:302] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c01eb21be64ed6b4b9a02bdc0433f9, ed4a396d62384eb48bfe8bcccf3955ea; no voters:
I20260928 05:23:37.118618 16128 raft_consensus.cc:2781] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:37.119028 16128 raft_consensus.cc:686] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 LEADER]: Becoming Leader. State: Replica: a0c01eb21be64ed6b4b9a02bdc0433f9, State: Running, Role: LEADER
I20260928 05:23:37.119553 16128 consensus_queue.cc:237] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } }
I20260928 05:23:37.124691 14833 catalog_manager.cc:5495] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:37.143046 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.138000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.8.124.193:33569)
0928 05:23:37.138186 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.138192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.138263 (+ 71us) server_negotiation.cc:407] Connection header received
0928 05:23:37.138503 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.138507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.138578 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.138699 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:37.139062 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.139747 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.141003 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.141411 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.141514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.142020 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:37.142515 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:37.142650 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.142721 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.142795 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:37.150838 15893 raft_consensus.cc:1261] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Refusing update from remote peer a0c01eb21be64ed6b4b9a02bdc0433f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:37.150836 16042 raft_consensus.cc:1261] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Refusing update from remote peer a0c01eb21be64ed6b4b9a02bdc0433f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:37.151749 16128 consensus_queue.cc:1035] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:37.152016 16132 consensus_queue.cc:1035] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:23:37.159333 15955 tablet.cc:2369] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:37.164650 16135 mvcc.cc:205] Tried to move back new op lower bound from 7334187078240571392 to 7334187078125273088. Current Snapshot: MvccSnapshot[applied={T|T < 7334187078240571392}]
I20260928 05:23:37.165261 16136 mvcc.cc:205] Tried to move back new op lower bound from 7334187078240571392 to 7334187078125273088. Current Snapshot: MvccSnapshot[applied={T|T < 7334187078240571392}]
I20260928 05:23:37.183918 14833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:37.184219 14833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:37.193532 14833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e8f1656733456a9c4d3d123be61db1: Sending DeleteTablet for 3 replicas of tablet 7879172661ba452f95be8d243fac8d8b
I20260928 05:23:37.194679 16022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7879172661ba452f95be8d243fac8d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:37 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260928 05:23:37.194676 15873 tablet_service.cc:1514] Processing DeleteTablet for tablet 7879172661ba452f95be8d243fac8d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:37 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:37534
I20260928 05:23:37.195874 15730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7879172661ba452f95be8d243fac8d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:37 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:58060
I20260928 05:23:37.196520 16145 tablet_replica.cc:330] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: stopping tablet replica
I20260928 05:23:37.196962 16145 raft_consensus.cc:2229] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:37.197445 16145 raft_consensus.cc:2258] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:37.199158 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 14801
I20260928 05:23:37.200181 14863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:37.205552 16144 tablet_replica.cc:330] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: stopping tablet replica
I20260928 05:23:37.206071 16144 raft_consensus.cc:2229] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:37.206746 16144 raft_consensus.cc:2258] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:37.206979 16145 ts_tablet_manager.cc:1905] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:37.211709 16144 ts_tablet_manager.cc:1905] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:37.211776 16146 tablet_replica.cc:330] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: stopping tablet replica
I20260928 05:23:37.212391 16146 raft_consensus.cc:2229] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:37.213143 16146 raft_consensus.cc:2258] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:37.215341 16146 ts_tablet_manager.cc:1905] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:37.215691 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.197917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.8.124.253:46597)
0928 05:23:37.198108 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.198113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.198131 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:37.198197 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.198203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.198279 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.198372 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:37.200564 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.201796 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.203847 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.204560 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.204659 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.208388 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:37.208397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:37.208402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:37.208448 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:37.212069 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.212955 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.212962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.212966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.213124 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.213712 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.213718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.213722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.214119 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:37.214352 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.215036 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.215297 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:37.222203 16145 ts_tablet_manager.cc:1918] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:37.222388 16145 log.cc:1196] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/7879172661ba452f95be8d243fac8d8b
I20260928 05:23:37.222915 16145 ts_tablet_manager.cc:1939] T 7879172661ba452f95be8d243fac8d8b P 23f5377cb1844d3980f2fa588288120a: Deleting consensus metadata
I20260928 05:23:37.223273 16144 ts_tablet_manager.cc:1918] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:37.223475 16144 log.cc:1196] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/7879172661ba452f95be8d243fac8d8b
I20260928 05:23:37.223989 16144 ts_tablet_manager.cc:1939] T 7879172661ba452f95be8d243fac8d8b P ed4a396d62384eb48bfe8bcccf3955ea: Deleting consensus metadata
I20260928 05:23:37.225196 16146 ts_tablet_manager.cc:1918] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:37.225696 14818 catalog_manager.cc:4841] TS ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645): tablet 7879172661ba452f95be8d243fac8d8b (table test-table [id=f997dc2683e445b4bb848824aeaaa6fd]) successfully deleted
I20260928 05:23:37.226265 16146 log.cc:1196] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/7879172661ba452f95be8d243fac8d8b
I20260928 05:23:37.227312 16146 ts_tablet_manager.cc:1939] T 7879172661ba452f95be8d243fac8d8b P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting consensus metadata
I20260928 05:23:37.233129 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.198892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.8.124.252:38333)
0928 05:23:37.199031 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.199036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.218254 (+ 19218us) server_negotiation.cc:407] Connection header received
0928 05:23:37.218506 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.218512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.218590 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.218656 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:37.220633 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.221776 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.223531 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.224141 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.224238 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.227726 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:37.227731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:37.227735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:37.227770 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:37.230615 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.231275 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.231281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.231284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.231419 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.231769 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.231774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.231777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.232060 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:37.232275 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.232585 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.232789 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 05:23:37.370285 14820 catalog_manager.cc:4841] TS 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565): tablet 7879172661ba452f95be8d243fac8d8b (table test-table [id=f997dc2683e445b4bb848824aeaaa6fd]) successfully deleted
I20260928 05:23:37.370431 14819 catalog_manager.cc:4841] TS a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569): tablet 7879172661ba452f95be8d243fac8d8b (table test-table [id=f997dc2683e445b4bb848824aeaaa6fd]) successfully deleted
W20260928 05:23:37.370536 14820 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.124.194:37565, user_credentials={real_user=slave}} blocked reactor thread for 145737us
I20260928 05:23:37.371207 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 14801
I20260928 05:23:37.419457 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.403687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.8.124.253:46597)
0928 05:23:37.403865 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.403870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.404821 (+ 951us) server_negotiation.cc:407] Connection header received
0928 05:23:37.405093 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.405098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.405195 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.405269 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:37.407786 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.408808 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.410131 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.410772 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.410896 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.413910 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:37.413916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:37.413921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:37.413960 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:37.416791 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.417476 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.417480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.417482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.417595 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.418045 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.418048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.418050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.418322 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:37.418547 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.418912 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.419117 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:23:37.421216 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:37.405095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.8.124.252:38333)
0928 05:23:37.405277 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:37.405283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:37.405300 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:37.405355 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:37.405360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:37.405444 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:37.405537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:37.408986 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.410071 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.411599 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:37.412175 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:37.412297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:37.414864 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:37.414870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:37.414875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:37.414913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:37.418201 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.419006 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.419011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.419014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.419161 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:37.419596 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:37.419600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:37.419603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:37.419962 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:37.420210 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:37.420722 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 05:23:37.420932 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:23:38.172886 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.171954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:38127)
0928 05:23:38.172516 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.172663 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":680,"threads_started":1}
W20260928 05:23:38.173307 15947 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:36501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.174111 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.173432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:51469)
0928 05:23:38.173841 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.173947 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":229,"threads_started":1}
W20260928 05:23:38.174437 16088 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:36501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.177517 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.176901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:59723)
0928 05:23:38.177279 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.177371 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":212,"threads_started":1}
W20260928 05:23:38.177847 15798 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:36501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.693430 16159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81e8f1656733456a9c4d3d123be61db1)
I20260928 05:23:38.693771 16160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81e8f1656733456a9c4d3d123be61db1)
I20260928 05:23:38.694024 16160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } }
I20260928 05:23:38.694077 16159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } }
I20260928 05:23:38.695003 16159 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501), 314195dd895c434b93957f375b8d8ae5 (127.8.124.252:38333)
I20260928 05:23:38.700635 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.695182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34080)
0928 05:23:38.700166 (+ 4984us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.700374 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4866,"threads_started":1}
W20260928 05:23:38.701575 15418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.704444 16160 leader_election.cc:288] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501), 4d6bb68f784046f990e6314c002b855e (127.8.124.253:46597)
I20260928 05:23:38.706305 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.695878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34096)
0928 05:23:38.705808 (+ 9930us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.706072 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9789,"thread_start_us":9937,"threads_started":1}
W20260928 05:23:38.706569 15605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.707916 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.707453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34104)
0928 05:23:38.707592 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.707733 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 05:23:38.708082 15418 leader_election.cc:334] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501): Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.709117 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.708745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34118)
0928 05:23:38.708900 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.708992 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:23:38.709340 15605 leader_election.cc:334] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501): Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.712543 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.695182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:38333 (local address 127.0.0.1:48594)
0928 05:23:38.700797 (+ 5615us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.700816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:38.700962 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:38.706542 (+ 5580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:38.706548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:38.706585 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:38.706886 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.706897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.708674 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.708680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.710651 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.710666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.711908 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.711912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.712038 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.712064 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:23:38.712134 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:23:38.712237 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5534,"mutex_wait_us":393,"thread_start_us":4330,"threads_started":1}
I20260928 05:23:38.712607 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.695439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.8.124.252:38333)
0928 05:23:38.701743 (+ 6304us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:38.701748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:38.701766 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:38.701832 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:38.701837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:38.701906 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:38.707040 (+ 5134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:38.707210 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.708492 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.710799 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.711760 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.711878 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.711997 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:38.712167 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 05:23:38.712287 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6184,"thread_start_us":6034,"threads_started":1}
I20260928 05:23:38.713186 15628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d6bb68f784046f990e6314c002b855e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "314195dd895c434b93957f375b8d8ae5" is_pre_election: true
I20260928 05:23:38.713488 15628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6bb68f784046f990e6314c002b855e in term 1.
I20260928 05:23:38.713719 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.702074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.8.124.253:46597)
0928 05:23:38.707006 (+ 4932us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:38.707012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:38.707036 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:23:38.707097 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:38.707102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:38.707177 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:38.708113 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:38.708419 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.709435 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.711517 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.712675 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.712786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.712880 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:38.713270 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 05:23:38.713398 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4807,"thread_start_us":4604,"threads_started":1}
I20260928 05:23:38.713789 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.702100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:46597 (local address 127.0.0.1:56428)
0928 05:23:38.706094 (+ 3994us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.706112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:38.706241 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:38.707609 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:38.707614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:38.707636 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:38.707931 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.707939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.709592 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.709624 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.711368 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.711379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.712951 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.712957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.713126 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.713150 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:23:38.713233 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:23:38.713351 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3868,"mutex_wait_us":3,"thread_start_us":3416,"threads_started":1}
I20260928 05:23:38.713974 15416 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 314195dd895c434b93957f375b8d8ae5, 4d6bb68f784046f990e6314c002b855e; no voters: 81e8f1656733456a9c4d3d123be61db1
I20260928 05:23:38.714310 16159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 05:23:38.714440 16159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 81e8f1656733456a9c4d3d123be61db1)
I20260928 05:23:38.714541 16159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:23:38.714464 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "314195dd895c434b93957f375b8d8ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4d6bb68f784046f990e6314c002b855e" is_pre_election: true
I20260928 05:23:38.716639 16159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } }
I20260928 05:23:38.717036 15441 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 314195dd895c434b93957f375b8d8ae5 in current term 2: Already voted for candidate 4d6bb68f784046f990e6314c002b855e in this term.
I20260928 05:23:38.717245 16159 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 election: Requested vote from peers 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501), 314195dd895c434b93957f375b8d8ae5 (127.8.124.252:38333)
I20260928 05:23:38.717783 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.717336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34124)
0928 05:23:38.717537 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.717628 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 05:23:38.717743 15628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d6bb68f784046f990e6314c002b855e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "314195dd895c434b93957f375b8d8ae5"
I20260928 05:23:38.717957 15628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:23:38.718649 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.718318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34134)
0928 05:23:38.718491 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.718564 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 05:23:38.718580 15603 leader_election.cc:302] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 314195dd895c434b93957f375b8d8ae5; no voters: 4d6bb68f784046f990e6314c002b855e, 81e8f1656733456a9c4d3d123be61db1
W20260928 05:23:38.718840 15418 leader_election.cc:334] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501): Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
I20260928 05:23:38.720436 15628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6bb68f784046f990e6314c002b855e in term 2.
I20260928 05:23:38.720626 16160 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:23:38.720834 15416 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 314195dd895c434b93957f375b8d8ae5, 4d6bb68f784046f990e6314c002b855e; no voters: 81e8f1656733456a9c4d3d123be61db1
I20260928 05:23:38.721230 16159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:23:38.721637 16159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [term 2 LEADER]: Becoming Leader. State: Replica: 4d6bb68f784046f990e6314c002b855e, State: Running, Role: LEADER
I20260928 05:23:38.722016 16159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } }
I20260928 05:23:38.723680 16169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d6bb68f784046f990e6314c002b855e. Latest consensus state: current_term: 2 leader_uuid: "4d6bb68f784046f990e6314c002b855e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:38.724073 16169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:38.724637 16171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:38.727970 16171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:38.728754 16171 catalog_manager.cc:1259] Loaded cluster ID: 5e13c503a25f44569fb8ec805868a783
I20260928 05:23:38.728825 16171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:38.730042 16171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:38.730794 16171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Loaded TSK: 0
I20260928 05:23:38.731542 16171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:23:38.742023 15430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:38.748535 15430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:38.752161 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.751795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34152)
0928 05:23:38.751967 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.752036 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 05:23:38.752198 15628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [term 2 FOLLOWER]: Refusing update from remote peer 4d6bb68f784046f990e6314c002b855e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 05:23:38.752833 16169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [LEADER]: Connected to new peer: Peer: permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 05:23:38.753131 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.752767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34154)
0928 05:23:38.752905 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.753007 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:23:38.753304 15418 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e -> Peer 81e8f1656733456a9c4d3d123be61db1 (127.8.124.254:36501): Couldn't send request to peer 81e8f1656733456a9c4d3d123be61db1. Status: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:23:38.755447 16160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d6bb68f784046f990e6314c002b855e. Latest consensus state: current_term: 2 leader_uuid: "4d6bb68f784046f990e6314c002b855e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:38.755694 16160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:38.756839 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d6bb68f784046f990e6314c002b855e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:38.757150 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:38.757455 16160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d6bb68f784046f990e6314c002b855e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "81e8f1656733456a9c4d3d123be61db1" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36501 } } peers { permanent_uuid: "4d6bb68f784046f990e6314c002b855e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 46597 } attrs { promote: false } } peers { permanent_uuid: "314195dd895c434b93957f375b8d8ae5" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 38333 } attrs { promote: false } } }
I20260928 05:23:38.757701 16160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314195dd895c434b93957f375b8d8ae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:23:38.772574 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.764688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33569 (local address 127.0.0.1:58080)
0928 05:23:38.765080 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.765091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:38.765157 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:38.765879 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:38.765885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:38.765915 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:38.766219 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.766230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.767829 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.767835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.770014 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.770030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.771764 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.771771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.771958 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.771999 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 05:23:38.772064 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 05:23:38.772209 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260928 05:23:38.773794 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.765093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.8.124.193:33569)
0928 05:23:38.765529 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:38.765535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:38.765552 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:38.765653 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:38.765658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:38.765728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:38.766368 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:38.766535 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.767665 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.770217 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.771508 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.773155 (+ 1647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.773270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:38.773331 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 05:23:38.773472 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":228,"threads_started":1}
I20260928 05:23:38.774444 15730 tablet_service.cc:1467] Processing CreateTablet for tablet fc61d6ead96243018a8a08b749138941 (DEFAULT_TABLE table=test-table [id=2dbc7c5710694b36866acb46ab5a287b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:38.774895 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc61d6ead96243018a8a08b749138941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:38.782835 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.765670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:38645 (local address 127.0.0.1:46804)
0928 05:23:38.772777 (+ 7107us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.772793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:38.772884 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:38.776849 (+ 3965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:38.776854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:38.776880 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:38.777034 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.777049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.778494 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.778499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.780378 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.780399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.782025 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.782032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.782229 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.782267 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:23:38.782344 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:23:38.782484 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7027,"mutex_wait_us":28,"thread_start_us":14289,"threads_started":1}
I20260928 05:23:38.783671 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.765800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.8.124.195:38645)
0928 05:23:38.776497 (+ 10697us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:38.776503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:38.776524 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:23:38.776595 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:38.776601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:38.776689 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:38.776793 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:38.777207 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.778314 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.780586 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.781852 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.783036 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.783143 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:38.783195 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 05:23:38.783352 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10563,"thread_start_us":11125,"threads_started":1}
I20260928 05:23:38.784409 16022 tablet_service.cc:1467] Processing CreateTablet for tablet fc61d6ead96243018a8a08b749138941 (DEFAULT_TABLE table=test-table [id=2dbc7c5710694b36866acb46ab5a287b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:38.784701 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.764414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:37565 (local address 127.0.0.1:37540)
0928 05:23:38.764748 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.764770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:38.764880 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:38.778477 (+ 13597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:38.778482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:38.778506 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:38.778679 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.778689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.779777 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.779782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.781612 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:38.781626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.784062 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.784068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:38.784248 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.784280 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:23:38.784333 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 05:23:38.784453 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1}
I20260928 05:23:38.784854 16022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc61d6ead96243018a8a08b749138941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:38.785535 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.764510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.8.124.194:37565)
0928 05:23:38.778093 (+ 13583us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:38.778097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:38.778112 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:38.778174 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:38.778179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:38.778245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:38.778824 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:38.778976 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.779656 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.781773 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:38.783883 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:38.784999 (+ 1116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:38.785091 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:38.785139 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 05:23:38.785275 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13491,"thread_start_us":13799,"threads_started":1}
I20260928 05:23:38.786108 16181 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Bootstrap starting.
I20260928 05:23:38.788118 15873 tablet_service.cc:1467] Processing CreateTablet for tablet fc61d6ead96243018a8a08b749138941 (DEFAULT_TABLE table=test-table [id=2dbc7c5710694b36866acb46ab5a287b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:38.788450 15873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc61d6ead96243018a8a08b749138941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:38.788653 16181 tablet_bootstrap.cc:654] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:38.790733 16181 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: No bootstrap required, opened a new log
I20260928 05:23:38.790946 16183 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Bootstrap starting.
I20260928 05:23:38.791095 16181 ts_tablet_manager.cc:1397] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 05:23:38.791688 16181 raft_consensus.cc:348] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.791865 16181 raft_consensus.cc:374] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:38.791949 16181 raft_consensus.cc:729] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c01eb21be64ed6b4b9a02bdc0433f9, State: Initialized, Role: FOLLOWER
I20260928 05:23:38.792155 16181 consensus_queue.cc:260] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.792526 16183 tablet_bootstrap.cc:654] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:38.793952 16181 ts_tablet_manager.cc:1428] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:38.794574 16183 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: No bootstrap required, opened a new log
I20260928 05:23:38.794692 16183 ts_tablet_manager.cc:1397] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:23:38.795432 16183 raft_consensus.cc:348] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.795660 16183 raft_consensus.cc:374] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:38.795747 16183 raft_consensus.cc:729] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4a396d62384eb48bfe8bcccf3955ea, State: Initialized, Role: FOLLOWER
I20260928 05:23:38.795953 16183 consensus_queue.cc:260] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.796319 16186 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Bootstrap starting.
I20260928 05:23:38.797673 16186 tablet_bootstrap.cc:654] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:38.799696 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.796548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:59929)
0928 05:23:38.799301 (+ 2753us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.799451 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2622,"thread_start_us":2530,"threads_started":1}
I20260928 05:23:38.800117 16186 tablet_bootstrap.cc:492] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: No bootstrap required, opened a new log
I20260928 05:23:38.800251 16186 ts_tablet_manager.cc:1397] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 05:23:38.800958 16183 ts_tablet_manager.cc:1428] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 05:23:38.800992 16186 raft_consensus.cc:348] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.801209 16186 raft_consensus.cc:374] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:38.801304 16186 raft_consensus.cc:729] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f5377cb1844d3980f2fa588288120a, State: Initialized, Role: FOLLOWER
I20260928 05:23:38.801651 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.800504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:35421)
0928 05:23:38.801128 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.801357 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":273,"threads_started":1}
I20260928 05:23:38.801517 16186 consensus_queue.cc:260] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:38.802640 16089 heartbeater.cc:503] Master 127.8.124.253:46597 was elected leader, sending a full tablet report...
I20260928 05:23:38.802942 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.802397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:60189)
0928 05:23:38.802529 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.802608 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:23:38.803375 16186 ts_tablet_manager.cc:1428] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 05:23:38.804517 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.803665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:53391)
0928 05:23:38.804113 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.804286 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"threads_started":1}
I20260928 05:23:38.804932 15948 heartbeater.cc:503] Master 127.8.124.253:46597 was elected leader, sending a full tablet report...
I20260928 05:23:38.805815 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:38.805468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:57613)
0928 05:23:38.805593 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:38.805692 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 05:23:38.816797 15806 tablet.cc:2369] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:23:38.940114 16090 tablet.cc:2369] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:39.005657 16184 raft_consensus.cc:482] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:39.005875 16184 raft_consensus.cc:504] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:39.006548 16184 leader_election.cc:288] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565), ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:39.007066 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc61d6ead96243018a8a08b749138941" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f5377cb1844d3980f2fa588288120a" is_pre_election: true
I20260928 05:23:39.007087 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc61d6ead96243018a8a08b749138941" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" is_pre_election: true
I20260928 05:23:39.007265 15893 raft_consensus.cc:2454] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 0.
I20260928 05:23:39.007308 16042 raft_consensus.cc:2454] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 0.
I20260928 05:23:39.007704 15686 leader_election.cc:302] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f5377cb1844d3980f2fa588288120a, a0c01eb21be64ed6b4b9a02bdc0433f9; no voters:
I20260928 05:23:39.007946 16184 raft_consensus.cc:2781] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:39.008049 16184 raft_consensus.cc:482] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:39.008134 16184 raft_consensus.cc:3037] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:39.010221 16184 raft_consensus.cc:504] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:39.010699 16184 leader_election.cc:288] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 election: Requested vote from peers 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565), ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645)
I20260928 05:23:39.011087 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc61d6ead96243018a8a08b749138941" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f5377cb1844d3980f2fa588288120a"
I20260928 05:23:39.011147 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc61d6ead96243018a8a08b749138941" candidate_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4a396d62384eb48bfe8bcccf3955ea"
I20260928 05:23:39.011263 15893 raft_consensus.cc:3037] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:39.011296 16042 raft_consensus.cc:3037] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:39.013111 16042 raft_consensus.cc:2454] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 1.
I20260928 05:23:39.013120 15893 raft_consensus.cc:2454] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c01eb21be64ed6b4b9a02bdc0433f9 in term 1.
I20260928 05:23:39.013485 15684 leader_election.cc:302] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23f5377cb1844d3980f2fa588288120a, a0c01eb21be64ed6b4b9a02bdc0433f9, ed4a396d62384eb48bfe8bcccf3955ea; no voters:
I20260928 05:23:39.013769 16184 raft_consensus.cc:2781] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:39.014158 16184 raft_consensus.cc:686] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 LEADER]: Becoming Leader. State: Replica: a0c01eb21be64ed6b4b9a02bdc0433f9, State: Running, Role: LEADER
I20260928 05:23:39.014420 16184 consensus_queue.cc:237] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } }
I20260928 05:23:39.014621 15803 heartbeater.cc:503] Master 127.8.124.253:46597 was elected leader, sending a full tablet report...
I20260928 05:23:39.015050 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.014674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:60405)
0928 05:23:39.014818 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.014900 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 05:23:39.017426 15429 catalog_manager.cc:5495] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c01eb21be64ed6b4b9a02bdc0433f9" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:39.069247 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.064821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.8.124.193:33569)
0928 05:23:39.064989 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:39.064994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:39.065015 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:23:39.065193 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:39.065198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:39.065263 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:39.065341 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:39.065658 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.066351 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.067601 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.068029 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.068121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:39.068557 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:39.068661 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:39.068725 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:39.068921 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 05:23:39.068986 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:23:39.072085 15893 raft_consensus.cc:1261] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Refusing update from remote peer a0c01eb21be64ed6b4b9a02bdc0433f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:39.072131 16042 raft_consensus.cc:1261] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Refusing update from remote peer a0c01eb21be64ed6b4b9a02bdc0433f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:39.072654 16184 consensus_queue.cc:1035] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f5377cb1844d3980f2fa588288120a" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:39.072906 16193 consensus_queue.cc:1035] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4a396d62384eb48bfe8bcccf3955ea" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:39.074754 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.074360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:45959)
0928 05:23:39.074517 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.074617 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:23:39.075799 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.075343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:32941)
0928 05:23:39.075508 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.075610 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:23:39.079092 16198 mvcc.cc:205] Tried to move back new op lower bound from 7334187086113832960 to 7334187085887016960. Current Snapshot: MvccSnapshot[applied={T|T < 7334187086113832960}]
I20260928 05:23:39.080464 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.079937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:42045)
0928 05:23:39.080158 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.080272 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 05:23:39.080638 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.080192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.195:52547)
0928 05:23:39.080413 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.080498 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 05:23:39.088459 15429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:39.088721 15429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:39.092058 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.091060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.193:59353)
0928 05:23:39.091777 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.091875 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600}
I20260928 05:23:39.095080 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.094561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.8.124.194:48055)
0928 05:23:39.094803 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.094889 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 05:23:39.099450 15429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d6bb68f784046f990e6314c002b855e: Sending DeleteTablet for 3 replicas of tablet fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.100445 16022 tablet_service.cc:1514] Processing DeleteTablet for tablet fc61d6ead96243018a8a08b749138941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:39 UTC) from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:46804
I20260928 05:23:39.100569 15873 tablet_service.cc:1514] Processing DeleteTablet for tablet fc61d6ead96243018a8a08b749138941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:39 UTC) from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:37540
I20260928 05:23:39.100927 15730 tablet_service.cc:1514] Processing DeleteTablet for tablet fc61d6ead96243018a8a08b749138941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:39 UTC) from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:58080
I20260928 05:23:39.101266 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet fc61d6ead96243018a8a08b749138941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:39 UTC) from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:37540
I20260928 05:23:39.101656 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet fc61d6ead96243018a8a08b749138941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:39 UTC) from {username='oryx', principal='oryx/127.8.124.253@KRBTEST.COM'} at 127.0.0.1:58080
I20260928 05:23:39.102514 16207 tablet_replica.cc:330] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: stopping tablet replica
I20260928 05:23:39.102720 16207 raft_consensus.cc:2229] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:39.102980 16207 raft_consensus.cc:2258] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:39.103722 16207 ts_tablet_manager.cc:1905] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:39.108137 16207 ts_tablet_manager.cc:1918] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:39.108265 16207 log.cc:1196] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.108745 16207 ts_tablet_manager.cc:1939] T fc61d6ead96243018a8a08b749138941 P 23f5377cb1844d3980f2fa588288120a: Deleting consensus metadata
I20260928 05:23:39.109928 15418 catalog_manager.cc:4841] TS 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565): tablet fc61d6ead96243018a8a08b749138941 (table test-table [id=2dbc7c5710694b36866acb46ab5a287b]) successfully deleted
W20260928 05:23:39.110350 15418 catalog_manager.cc:4808] TS 23f5377cb1844d3980f2fa588288120a (127.8.124.194:37565): delete failed for tablet fc61d6ead96243018a8a08b749138941 because the tablet was not found. No further retry: Not found: Tablet not found: fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.111114 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15590
I20260928 05:23:39.115231 16206 tablet_replica.cc:330] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: stopping tablet replica
I20260928 05:23:39.115448 16206 raft_consensus.cc:2229] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:39.115744 16206 raft_consensus.cc:2258] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:39.116868 16206 ts_tablet_manager.cc:1905] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:39.118875 16208 tablet_replica.cc:330] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: stopping tablet replica
I20260928 05:23:39.119055 16208 raft_consensus.cc:2229] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:39.119491 16208 raft_consensus.cc:2258] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:39.120329 16208 ts_tablet_manager.cc:1905] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:39.121876 16206 ts_tablet_manager.cc:1918] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:39.121975 16206 log.cc:1196] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.122344 16206 ts_tablet_manager.cc:1939] T fc61d6ead96243018a8a08b749138941 P ed4a396d62384eb48bfe8bcccf3955ea: Deleting consensus metadata
I20260928 05:23:39.123456 15416 catalog_manager.cc:4841] TS ed4a396d62384eb48bfe8bcccf3955ea (127.8.124.195:38645): tablet fc61d6ead96243018a8a08b749138941 (table test-table [id=2dbc7c5710694b36866acb46ab5a287b]) successfully deleted
I20260928 05:23:39.125232 16208 ts_tablet_manager.cc:1918] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:39.125360 16208 log.cc:1196] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.125811 16208 ts_tablet_manager.cc:1939] T fc61d6ead96243018a8a08b749138941 P a0c01eb21be64ed6b4b9a02bdc0433f9: Deleting consensus metadata
I20260928 05:23:39.126772 15417 catalog_manager.cc:4841] TS a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569): tablet fc61d6ead96243018a8a08b749138941 (table test-table [id=2dbc7c5710694b36866acb46ab5a287b]) successfully deleted
W20260928 05:23:39.127148 15417 catalog_manager.cc:4808] TS a0c01eb21be64ed6b4b9a02bdc0433f9 (127.8.124.193:33569): delete failed for tablet fc61d6ead96243018a8a08b749138941 because the tablet was not found. No further retry: Not found: Tablet not found: fc61d6ead96243018a8a08b749138941
I20260928 05:23:39.130471 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.111885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.8.124.193:33569)
0928 05:23:39.112072 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:39.112079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:39.114014 (+ 1935us) server_negotiation.cc:407] Connection header received
0928 05:23:39.114167 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:39.114172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:39.114231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:39.114321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:39.115498 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.116508 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.117878 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.118415 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.118528 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:39.124612 (+ 6084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:39.124626 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:39.124643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:39.124674 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:39.127775 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.128549 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.128556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.128559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.128683 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.129082 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.129085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.129088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.129459 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:39.129707 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:39.129998 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:23:39.130159 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:23:39.131090 15792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:39.251214 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15590
I20260928 05:23:39.269178 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15810
I20260928 05:23:39.281095 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.269871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.8.124.194:37565)
0928 05:23:39.270059 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:39.270063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:39.270081 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:39.270183 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:39.270187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:39.270246 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:39.270341 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:39.271586 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.272244 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.273419 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.273910 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.274001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:39.276251 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:39.276263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:39.276279 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:39.276305 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:39.278917 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.279488 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.279494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.279497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.279601 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.279857 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.279860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.279861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.280191 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:39.280398 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:39.280620 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 05:23:39.280766 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:39.281733 15941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:39.295670 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.295027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34162)
0928 05:23:39.295434 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.295513 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":193,"threads_started":1}
I20260928 05:23:39.298542 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.298193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:36501 (local address 127.0.0.1:34168)
0928 05:23:39.298330 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:39.298428 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:23:39.373721 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15810
I20260928 05:23:39.388156 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15959
I20260928 05:23:39.400331 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:39.388848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.8.124.195:38645)
0928 05:23:39.389253 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:39.389258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:39.389273 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:39.389322 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:39.389325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:39.389388 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:39.389473 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:39.390753 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.391441 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.392598 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:39.392997 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:39.393083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:39.395404 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:39.395417 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:39.395433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:39.395464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:39.398128 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.398721 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.398727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.398729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.398834 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:39.399138 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:39.399141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:39.399143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:39.399457 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:39.399656 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:39.399874 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 05:23:39.400032 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":209,"threads_started":1}
I20260928 05:23:39.400925 16082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:39.491060 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15959
I20260928 05:23:39.504777 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15294
I20260928 05:23:39.505749 15461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:39.622529 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15294
I20260928 05:23:39.639001 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 15395
I20260928 05:23:39.639961 15648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:39.749620 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 15395
2026-09-28T05:23:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8845 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:39 dist-test-slave-515x krb5kdc[16220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:39 dist-test-slave-515x krb5kdc[16220](info): set up 2 sockets
Sep 28 05:23:39 dist-test-slave-515x krb5kdc[16220](info): commencing operation
krb5kdc: starting...
W20260928 05:23:40.754231 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:40 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab.
Entry for principal oryx/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab.
I20260928 05:23:40.952632 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37625
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab
--principal=oryx/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:41.260532 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:41.260807 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:41.260896 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:41.267917 16233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:41.268015 16233 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:23:41.268062 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:41.268119 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:41.268167 16233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:41.268224 16233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:41.277719 16233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:37625
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.254.keytab
--principal=oryx/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:41.278908 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:41.280160 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:41.291359 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:41.291396 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:41.291422 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:41.292101 16233 server_base.cc:1034] running on GCE node
I20260928 05:23:41.292820 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:41.294210 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:41.295408 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790573021295334 us; error 46 us; skew 500 ppm
I20260928 05:23:41.295710 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:41 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573021, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:41.298558 16233 init.cc:377] Logged in from keytab as oryx/127.8.124.254@KRBTEST.COM (short username oryx)
I20260928 05:23:41.299893 16233 webserver.cc:460] Webserver started at http://127.8.124.254:37113/ using document root <none> and password file <none>
I20260928 05:23:41.300567 16233 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:41.300662 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:41.300961 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:41.303828 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "6798823038ad4c15b9ef21e20aaf7633"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "9b023be27d1354dea6100d38c6030902"
server_key_iv: "a2716a7283ad5757101d5d9aee885ee5"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.304582 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "6798823038ad4c15b9ef21e20aaf7633"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "9b023be27d1354dea6100d38c6030902"
server_key_iv: "a2716a7283ad5757101d5d9aee885ee5"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.309316 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 05:23:41.312493 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:41.314112 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:23:41.314270 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "6798823038ad4c15b9ef21e20aaf7633"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "9b023be27d1354dea6100d38c6030902"
server_key_iv: "a2716a7283ad5757101d5d9aee885ee5"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.314493 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:41.330540 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:41.334446 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:41.334661 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:41.353612 16233 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:37625
I20260928 05:23:41.353642 16309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:37625 every 8 connection(s)
I20260928 05:23:41.354907 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:41.358433 16310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:41.360571 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 16233
I20260928 05:23:41.360765 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:41.361506 8691 external_mini_cluster.cc:1428] Setting key b12811c857397ef48c3a2712ec292328
I20260928 05:23:41.370157 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Bootstrap starting.
I20260928 05:23:41.373677 16310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:41.374897 16310 log.cc:824] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:23:41 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:41.377507 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: No bootstrap required, opened a new log
I20260928 05:23:41.382997 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:41.363460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.8.124.254:37625)
0928 05:23:41.364292 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:41.364308 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:41.364351 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:23:41.366027 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:41.366049 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:41.366406 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:41.366732 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:41.368349 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.369590 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.371310 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.371992 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.372138 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:41.376293 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:41.376301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:41.376315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:41.376348 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:41.379224 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.379837 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.379847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.379853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.380023 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.380367 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.380370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.380372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.380887 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:41.381165 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:41.381588 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 05:23:41.382007 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":383,"threads_started":1}
I20260928 05:23:41.385192 16310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } }
I20260928 05:23:41.385689 16310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:41.385783 16310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6798823038ad4c15b9ef21e20aaf7633, State: Initialized, Role: FOLLOWER
I20260928 05:23:41.386322 16310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } }
I20260928 05:23:41.386525 16310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:41.386615 16310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:41.386721 16310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:41.388752 16310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } }
I20260928 05:23:41.389158 16310 leader_election.cc:302] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6798823038ad4c15b9ef21e20aaf7633; no voters:
I20260928 05:23:41.389707 16310 leader_election.cc:288] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:41.389817 16315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:41.390779 16315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [term 1 LEADER]: Becoming Leader. State: Replica: 6798823038ad4c15b9ef21e20aaf7633, State: Running, Role: LEADER
I20260928 05:23:41.391255 16315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } }
I20260928 05:23:41.391801 16310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:41.393677 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6798823038ad4c15b9ef21e20aaf7633. Latest consensus state: current_term: 1 leader_uuid: "6798823038ad4c15b9ef21e20aaf7633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } } }
I20260928 05:23:41.393628 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6798823038ad4c15b9ef21e20aaf7633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6798823038ad4c15b9ef21e20aaf7633" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37625 } } }
I20260928 05:23:41.394013 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:41.394009 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633 [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:41.395432 16322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:41.400609 16322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:41.409329 16322 catalog_manager.cc:1347] Generated new cluster ID: 738595a1beb04e50a914f6c5ddaca0ad
I20260928 05:23:41.409449 16322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:41.443096 16322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:41.444357 16322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:41.451131 16322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Generated new TSK 0
I20260928 05:23:41.451902 16322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
I20260928 05:23:41.507107 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37625
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:41.814769 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:41.815027 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:41.815142 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:41.822162 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:41.822256 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:41.822372 16336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:23:41.822468 16336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:41.831835 16336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:41.833019 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:41.834298 16336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:41.846151 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:41.846223 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:41.846202 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:41.846922 16336 server_base.cc:1034] running on GCE node
I20260928 05:23:41.847414 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:41.848184 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:41.849378 16336 hybrid_clock.cc:648] HybridClock initialized: now 1790573021849313 us; error 41 us; skew 500 ppm
I20260928 05:23:41.849681 16336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:41 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573021, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:41.852579 16336 init.cc:377] Logged in from keytab as oryx/127.8.124.193@KRBTEST.COM (short username oryx)
I20260928 05:23:41.853855 16336 webserver.cc:460] Webserver started at http://127.8.124.193:35767/ using document root <none> and password file <none>
I20260928 05:23:41.854552 16336 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:41.854645 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:41.854974 16336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:41.857808 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "df8dd992665445f2b51ab503c874cea0"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "d33932bf91a28248af84c14afd44186a"
server_key_iv: "73d2da075b0ab7c798e91e45cd4a5f1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.858547 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "df8dd992665445f2b51ab503c874cea0"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "d33932bf91a28248af84c14afd44186a"
server_key_iv: "73d2da075b0ab7c798e91e45cd4a5f1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.863507 16336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:23:41.866745 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:41.868016 16336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:23:41.868160 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "df8dd992665445f2b51ab503c874cea0"
format_stamp: "Formatted at 2026-09-28 05:23:41 on dist-test-slave-515x"
server_key: "d33932bf91a28248af84c14afd44186a"
server_key_iv: "73d2da075b0ab7c798e91e45cd4a5f1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:41.868343 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:41.891458 16336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:41.895668 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:41.895900 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:41.898516 16336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:41.898597 16336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:41.898715 16336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:41.898772 16336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:23:41 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573021, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:41.920653 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:41.898940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:52151 (local address 127.8.124.254:37625)
0928 05:23:41.901471 (+ 2531us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:41.901477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:41.901498 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:23:41.903562 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:41.903568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:41.903651 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:41.903768 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:41.906243 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.906930 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.909629 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.910134 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.910258 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:41.914873 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:41.914879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:41.914883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:41.914924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:41.917341 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.918358 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.918364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.918367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.918518 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.918995 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.918999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.919003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.919292 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:41.919415 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:41.920084 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 05:23:41.920287 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":2436,"threads_started":1}
I20260928 05:23:41.921877 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:41.899738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.193:52151)
0928 05:23:41.900679 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:41.900726 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:41.903206 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:41.904065 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:41.904081 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:41.904919 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:41.906031 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:41.906074 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.907088 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.907094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:41.909454 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:41.909467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.910420 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.910428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:41.910747 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:41.911522 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:41.911544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:41.914572 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:41.917477 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:41.917488 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:41.917507 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:41.918222 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:41.918643 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:41.918647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:41.918651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:41.918889 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:41.919408 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:41.919416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:41.919749 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 05:23:41.919986 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 05:23:41.920483 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":380,"threads_started":1}
I20260928 05:23:41.941368 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:41.927920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:47107 (local address 127.8.124.254:37625)
0928 05:23:41.928106 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:41.928112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:41.928644 (+ 532us) server_negotiation.cc:407] Connection header received
0928 05:23:41.928801 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:41.928804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:41.928860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:41.928933 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:41.930244 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.930973 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.932922 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.933305 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.933396 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:41.935576 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:41.935581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:41.935584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:41.935613 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:41.938390 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.939350 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.939356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.939360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.939548 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:41.939953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:41.939958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:41.939962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:41.940297 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:41.940456 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:41.940814 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 05:23:41.941046 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:23:41.941418 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:41.927924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.193:47107)
0928 05:23:41.928534 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:41.928556 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:41.928703 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:41.928974 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:41.928979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:41.929586 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:41.930088 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:41.930105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.931094 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.931100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:41.932781 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:41.932797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:41.933477 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:41.933484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:41.933696 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:41.934321 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:41.934343 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:41.935400 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:41.938555 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:41.938562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:41.938567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:41.939210 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:41.939659 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:41.939664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:41.939674 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:41.939835 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:41.940401 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:41.940404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:41.940571 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 05:23:41.940698 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 05:23:41.940930 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":300,"threads_started":1}
I20260928 05:23:41.948001 16336 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:45039
I20260928 05:23:41.948030 16471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:45039 every 8 connection(s)
I20260928 05:23:41.949134 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:41.951123 16472 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37625
I20260928 05:23:41.951336 16472 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:41.951915 16472 heartbeater.cc:511] Master 127.8.124.254:37625 requested a full tablet report, sending...
I20260928 05:23:41.953930 16265 ts_manager.cc:194] Registered new tserver with Master: df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039)
I20260928 05:23:41.955410 16265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:52151
I20260928 05:23:41.956727 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 16336
I20260928 05:23:41.956915 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:41.957713 8691 external_mini_cluster.cc:1428] Setting key f9131895bb88a86285aeeb60d76e3240
WARNING: no policy specified for oryx/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
I20260928 05:23:41.998257 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37625
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:42.314558 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:42.314823 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:42.314957 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:42.323031 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:42.323154 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:42.323287 16476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:23:42.323413 16476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:42.333474 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:42.334813 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:42.336095 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:42.348312 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:42.348356 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:42.348764 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:42.348860 16476 server_base.cc:1034] running on GCE node
I20260928 05:23:42.349437 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:42.350345 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:42.351586 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790573022351511 us; error 43 us; skew 500 ppm
I20260928 05:23:42.351871 16476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:42.354910 16476 init.cc:377] Logged in from keytab as oryx/127.8.124.194@KRBTEST.COM (short username oryx)
I20260928 05:23:42.356317 16476 webserver.cc:460] Webserver started at http://127.8.124.194:33329/ using document root <none> and password file <none>
I20260928 05:23:42.356930 16476 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:42.357008 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:42.357290 16476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:42.360070 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "214b60f54ab44fdab111bb78c4ab87a8"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "b7bfc2d184fc809092c8153519c23661"
server_key_iv: "742e9d60c44ca7b9fe17d9221bfb359e"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.360790 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "214b60f54ab44fdab111bb78c4ab87a8"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "b7bfc2d184fc809092c8153519c23661"
server_key_iv: "742e9d60c44ca7b9fe17d9221bfb359e"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.365832 16476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:23:42.369246 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:42.370534 16476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:42.370666 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "214b60f54ab44fdab111bb78c4ab87a8"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "b7bfc2d184fc809092c8153519c23661"
server_key_iv: "742e9d60c44ca7b9fe17d9221bfb359e"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.370862 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:42.391901 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:42.396239 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:42.396476 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:42.399086 16476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:42.399163 16476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:42.399271 16476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:42.399322 16476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:42.422151 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.399701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:38651 (local address 127.8.124.254:37625)
0928 05:23:42.399874 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:42.399880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:42.401374 (+ 1494us) server_negotiation.cc:407] Connection header received
0928 05:23:42.403454 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:42.403461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:42.403550 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:42.403623 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:42.405987 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.406992 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.409796 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.410321 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.410467 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.415500 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:42.415506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:42.415511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:42.415576 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:42.418447 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.419449 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.419455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.419459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.419592 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.420047 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.420052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.420056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.420412 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:42.420552 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:42.421485 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 05:23:42.421765 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:23:42.422971 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.400327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.194:38651)
0928 05:23:42.401201 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:42.401248 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:42.403108 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:42.404183 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:42.404198 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:42.404920 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:42.405795 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.405830 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.407147 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.407153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.409629 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.409650 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.410772 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.410780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.411072 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.411867 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:42.411911 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:42.415185 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:42.418584 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.418594 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.418617 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.419315 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.419703 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.419708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.419712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.419933 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.420523 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:42.420534 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:42.421032 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 05:23:42.421372 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 05:23:42.421801 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":342,"threads_started":1}
I20260928 05:23:42.440795 16476 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:44371
I20260928 05:23:42.440816 16610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:44371 every 8 connection(s)
I20260928 05:23:42.442119 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.428451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:36835 (local address 127.8.124.254:37625)
0928 05:23:42.428594 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:42.428598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:42.429146 (+ 548us) server_negotiation.cc:407] Connection header received
0928 05:23:42.429264 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:42.429269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:42.429347 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:42.429445 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:42.430662 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.431729 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.433405 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.434048 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.434171 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.436552 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:42.436559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:42.436563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:42.436609 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:42.439497 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.440270 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.440274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.440278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.440395 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.440731 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.440736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.440739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.441091 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:42.441216 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:42.441584 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 05:23:42.441804 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:23:42.442209 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:42.442243 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.428507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.194:36835)
0928 05:23:42.429049 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:42.429067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:42.429169 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:42.429495 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:42.429498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:42.430044 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:42.430502 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.430520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.431882 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.431888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.433273 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.433284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.434418 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.434425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.434622 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.435320 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:42.435344 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:42.436362 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:42.439649 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.439654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.439658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.440173 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.440506 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.440511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.440515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.440637 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.441194 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:42.441199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:42.441364 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 05:23:42.441491 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 05:23:42.441824 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260928 05:23:42.445209 16611 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37625
I20260928 05:23:42.445406 16611 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:42.446025 16611 heartbeater.cc:511] Master 127.8.124.254:37625 requested a full tablet report, sending...
I20260928 05:23:42.446355 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 16476
I20260928 05:23:42.446516 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:42.446956 8691 external_mini_cluster.cc:1428] Setting key 9d95e8fbaed6aabab8e23f1f33e81c4b
I20260928 05:23:42.447245 16265 ts_manager.cc:194] Registered new tserver with Master: 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371)
I20260928 05:23:42.448359 16265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:38651
WARNING: no policy specified for oryx/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
I20260928 05:23:42.490510 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37625
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:42.800637 16615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:42.800913 16615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:42.801039 16615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:42.808264 16615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:42.808391 16615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:42.808516 16615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:23:42.808624 16615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:42.818251 16615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.124.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:42.819604 16615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:42.820870 16615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:42.833315 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:42.833280 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:42.833351 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:42.834633 16615 server_base.cc:1034] running on GCE node
I20260928 05:23:42.835213 16615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:42.836140 16615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:42.837443 16615 hybrid_clock.cc:648] HybridClock initialized: now 1790573022837354 us; error 41 us; skew 500 ppm
I20260928 05:23:42.837893 16615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:42.840786 16615 init.cc:377] Logged in from keytab as oryx/127.8.124.195@KRBTEST.COM (short username oryx)
I20260928 05:23:42.842180 16615 webserver.cc:460] Webserver started at http://127.8.124.195:46135/ using document root <none> and password file <none>
I20260928 05:23:42.842825 16615 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:42.842916 16615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:42.843230 16615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:42.846086 16615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "e19018bf902a46ddb2b74c5f79e510f2"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "4adbbc35a80f23d3c0865e43752730d3"
server_key_iv: "ab2a24caf7e1567211d4809d21e5adb8"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.846776 16615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "e19018bf902a46ddb2b74c5f79e510f2"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "4adbbc35a80f23d3c0865e43752730d3"
server_key_iv: "ab2a24caf7e1567211d4809d21e5adb8"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.851609 16615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:23:42.854987 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:42.856284 16615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:23:42.856478 16615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "e19018bf902a46ddb2b74c5f79e510f2"
format_stamp: "Formatted at 2026-09-28 05:23:42 on dist-test-slave-515x"
server_key: "4adbbc35a80f23d3c0865e43752730d3"
server_key_iv: "ab2a24caf7e1567211d4809d21e5adb8"
server_key_version: "encryptionkey@0"
I20260928 05:23:42.856683 16615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:42.875906 16615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:42.879882 16615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:42.880122 16615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:42.882682 16615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:42.882784 16615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:42.882907 16615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:42.882974 16615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:42.903220 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.883281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:51871 (local address 127.8.124.254:37625)
0928 05:23:42.883474 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:42.883481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:42.885066 (+ 1585us) server_negotiation.cc:407] Connection header received
0928 05:23:42.887450 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:42.887456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:42.887550 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:42.887633 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:42.889587 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.890744 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.893219 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.893806 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.893932 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.897741 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:42.897746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:42.897749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:42.897784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:42.900271 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.900942 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.900947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.900951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.901078 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.901460 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.901465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.901469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.901794 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:42.901919 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:42.902667 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 05:23:42.902889 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:23:42.904230 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.883963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.195:51871)
0928 05:23:42.884864 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:42.884933 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:42.887145 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:42.887914 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:42.887928 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:42.888630 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:42.889377 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.889418 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.890937 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.890945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.893064 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.893089 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.894038 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.894045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.894268 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.894987 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:42.895008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:42.897524 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:42.900410 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.900420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.900439 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.900837 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.901175 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.901180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.901184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.901366 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.901910 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:42.901918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:42.902292 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 05:23:42.902537 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 05:23:42.903034 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260928 05:23:42.923511 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.910326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:56443 (local address 127.8.124.254:37625)
0928 05:23:42.910548 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:42.910555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:42.910649 (+ 94us) server_negotiation.cc:407] Connection header received
0928 05:23:42.910802 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:42.910809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:42.910890 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:42.911000 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:42.911816 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.912958 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.914532 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.915063 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.915136 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.917788 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:42.917793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:42.917796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:42.917829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:42.920893 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.921703 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.921708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.921712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.921835 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:42.922168 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:42.922172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:42.922176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:42.922428 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:42.922558 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:42.922968 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 05:23:42.923169 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:23:42.923619 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:42.909906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.8.124.195:56443)
0928 05:23:42.910551 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:42.910571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:42.910704 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:42.911041 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:42.911045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:42.911421 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:42.911702 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.911711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.913089 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.913092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.914350 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:42.914367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.915371 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.915377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:42.915806 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:42.916533 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:42.916558 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:42.917574 (+ 1016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:42.921014 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.921020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.921025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.921574 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.921933 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:42.921937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:42.921941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:42.922066 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:42.922536 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:42.922541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:42.922707 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 05:23:42.922837 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:23:42.923145 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":474,"threads_started":1}
I20260928 05:23:42.939287 16615 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:41043
I20260928 05:23:42.939314 16749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:41043 every 8 connection(s)
I20260928 05:23:42.940464 16615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:42.942536 16750 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37625
I20260928 05:23:42.942736 16750 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:42.943356 16750 heartbeater.cc:511] Master 127.8.124.254:37625 requested a full tablet report, sending...
I20260928 05:23:42.944576 16265 ts_manager.cc:194] Registered new tserver with Master: e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:42.945353 16265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:51871
I20260928 05:23:42.949271 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 16615
I20260928 05:23:42.949406 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:42.949873 8691 external_mini_cluster.cc:1428] Setting key 60f1961f822509f9eaac74695f0d1af9
I20260928 05:23:42.952611 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:23:42.958455 16472 heartbeater.cc:503] Master 127.8.124.254:37625 was elected leader, sending a full tablet report...
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:42 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:43.451050 16611 heartbeater.cc:503] Master 127.8.124.254:37625 was elected leader, sending a full tablet report...
I20260928 05:23:43.947615 16750 heartbeater.cc:503] Master 127.8.124.254:37625 was elected leader, sending a full tablet report...
W20260928 05:23:43.958699 16758 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.966s user 0.009s sys 0.003s
W20260928 05:23:43.959826 16360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:42.987926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.8.124.254:37625)
0928 05:23:42.988092 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:42.988096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:42.988350 (+ 254us) server_negotiation.cc:407] Connection header received
0928 05:23:42.988508 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:42.988512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:42.988575 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:42.988665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:42.989891 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.990626 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.991820 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:42.992224 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:42.992313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:43.959519 (+967206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49960: BlockingRecv error: recv got EOF from 127.0.0.1:49960 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:23:43 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:43.975560 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.963040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.8.124.254:37625)
0928 05:23:43.963198 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:43.963204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:43.963522 (+ 318us) server_negotiation.cc:407] Connection header received
0928 05:23:43.963679 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:43.963684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:43.963773 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:43.963849 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:43.965020 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:43.965809 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:43.966972 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:43.967369 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:43.967483 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:43.970849 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:43.970854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:43.970857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:43.970888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:43.973127 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:43.973779 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:43.973784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:43.973786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:43.973904 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:43.974242 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:43.974247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:43.974250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:43.974529 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:43.974869 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:43.975053 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:23:43.975263 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:23:43.979239 16265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:43.985899 16265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:44.007833 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.996712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44371 (local address 127.0.0.1:39472)
0928 05:23:43.997644 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:43.997684 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:43.997862 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:43.999914 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:43.999918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:43.999938 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.000333 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.000343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.002199 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.002204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.004947 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.004959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.006844 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.006851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.007036 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.007092 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.007200 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.007386 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":133,"thread_start_us":1250,"threads_started":1}
I20260928 05:23:44.008373 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.997193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.8.124.193:45039)
0928 05:23:43.999287 (+ 2094us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:43.999293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:43.999311 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:43.999372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:43.999376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:43.999527 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:43.999686 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.000424 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.001999 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.005123 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.006621 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.006782 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.006994 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.007941 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.008119 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1974,"thread_start_us":1849,"threads_started":1}
I20260928 05:23:44.009239 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.997146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.8.124.194:44371)
0928 05:23:43.998382 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:43.998391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:43.998417 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:23:43.998516 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:43.998524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:43.998783 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.000511 (+ 1728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.000708 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.002081 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.005121 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.006621 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.008581 (+ 1960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.008787 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.008851 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.008988 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":1241,"threads_started":1}
I20260928 05:23:44.007853 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.996901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:45039 (local address 127.0.0.1:60398)
0928 05:23:43.998204 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:43.998221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:43.998318 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:43.999713 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:43.999720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:43.999747 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.000199 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.000215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.002171 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.002177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.004945 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.004959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.007138 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.007144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.007287 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.007316 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.007384 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.007485 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":180,"thread_start_us":310,"threads_started":1}
I20260928 05:23:44.013799 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 (DEFAULT_TABLE table=test-table [id=fa716efd04f744b699dbd9e2a301650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.013799 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 (DEFAULT_TABLE table=test-table [id=fa716efd04f744b699dbd9e2a301650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.015239 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14836daf25d7423b90532a1f6b2f7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.015233 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14836daf25d7423b90532a1f6b2f7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.015364 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.998289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:41043 (local address 127.0.0.1:48432)
0928 05:23:44.008309 (+ 10020us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.008328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.008424 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.010303 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.010308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.010333 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.010517 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.010527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.011963 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.011967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.013403 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.013421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.014671 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.014676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.014857 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.014888 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.014961 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.015098 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9933,"thread_start_us":11605,"threads_started":1}
I20260928 05:23:44.015413 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:43.998760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.8.124.195:41043)
0928 05:23:44.009773 (+ 11013us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.009779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.009802 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:23:44.009874 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.009880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.010074 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.010301 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.010657 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.011819 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.013555 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.014521 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.014645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.014832 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.015009 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.015163 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10882,"thread_start_us":10729,"threads_started":1}
I20260928 05:23:44.019055 16685 tablet_service.cc:1467] Processing CreateTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 (DEFAULT_TABLE table=test-table [id=fa716efd04f744b699dbd9e2a301650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.020408 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14836daf25d7423b90532a1f6b2f7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.029019 16776 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Bootstrap starting.
I20260928 05:23:44.030854 16775 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Bootstrap starting.
I20260928 05:23:44.032685 16776 tablet_bootstrap.cc:654] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.034096 16776 log.cc:824] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:44.034140 16775 tablet_bootstrap.cc:654] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.034693 16777 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Bootstrap starting.
I20260928 05:23:44.035130 16775 log.cc:824] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:44.037262 16777 tablet_bootstrap.cc:654] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.037472 16775 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: No bootstrap required, opened a new log
I20260928 05:23:44.037738 16775 ts_tablet_manager.cc:1397] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260928 05:23:44.038228 16777 log.cc:824] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:44.039333 16776 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: No bootstrap required, opened a new log
I20260928 05:23:44.039585 16776 ts_tablet_manager.cc:1397] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260928 05:23:44.040354 16777 tablet_bootstrap.cc:492] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: No bootstrap required, opened a new log
I20260928 05:23:44.040562 16777 ts_tablet_manager.cc:1397] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 05:23:44.047465 16777 raft_consensus.cc:348] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.047785 16777 raft_consensus.cc:374] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.047550 16775 raft_consensus.cc:348] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.047901 16777 raft_consensus.cc:729] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.047993 16775 raft_consensus.cc:374] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.048084 16775 raft_consensus.cc:729] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.048468 16777 consensus_queue.cc:260] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.048652 16775 consensus_queue.cc:260] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.049260 16776 raft_consensus.cc:348] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.049731 16776 raft_consensus.cc:374] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.049847 16776 raft_consensus.cc:729] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.050115 16777 ts_tablet_manager.cc:1428] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Time spent starting tablet: real 0.009s user 0.009s sys 0.002s
I20260928 05:23:44.050508 16776 consensus_queue.cc:260] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.051664 16775 ts_tablet_manager.cc:1428] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260928 05:23:44.054116 16776 ts_tablet_manager.cc:1428] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260928 05:23:44.192800 16751 tablet.cc:2369] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:23:44.194768 16612 tablet.cc:2369] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:23:44.201912 16473 tablet.cc:2369] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:23:44.202448 16782 raft_consensus.cc:482] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:44.202678 16782 raft_consensus.cc:504] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.204214 16782 leader_election.cc:288] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.210362 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.204660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:37613 (local address 127.8.124.194:44371)
0928 05:23:44.204847 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.204853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.205295 (+ 442us) server_negotiation.cc:407] Connection header received
0928 05:23:44.205526 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.205532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.205637 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.205698 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.206348 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.207108 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.208813 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.209542 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.209715 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.209786 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.210013 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.210129 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:23:44.210515 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.204569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44371 (local address 127.8.124.193:37613)
0928 05:23:44.205177 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.205192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.205345 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.205851 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.205853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.205872 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.206227 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.206236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.207210 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.207215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.208680 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.208691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.209750 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.209755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.209883 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.209905 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.209980 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.210118 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1,"spinlock_wait_cycles":11648,"thread_start_us":333,"threads_started":1}
I20260928 05:23:44.210659 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.205420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:35157 (local address 127.8.124.195:41043)
0928 05:23:44.205627 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.205632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.205924 (+ 292us) server_negotiation.cc:407] Connection header received
0928 05:23:44.206100 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.206104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.206162 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.206231 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.206743 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.207452 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.209097 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.209855 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.210000 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.210068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.210315 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.210446 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:23:44.210733 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.205277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:41043 (local address 127.8.124.193:35157)
0928 05:23:44.205832 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.205845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.205951 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.206258 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.206262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.206284 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.206602 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.206610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.207573 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.207577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.208959 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.208970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.210051 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.210057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.210178 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.210200 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.210280 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.210411 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":251,"threads_started":1}
I20260928 05:23:44.211541 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14836daf25d7423b90532a1f6b2f7cf8" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8" is_pre_election: true
I20260928 05:23:44.211546 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14836daf25d7423b90532a1f6b2f7cf8" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2" is_pre_election: true
I20260928 05:23:44.211971 16705 raft_consensus.cc:2454] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:44.211974 16561 raft_consensus.cc:2454] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:44.212491 16352 leader_election.cc:302] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df8dd992665445f2b51ab503c874cea0, e19018bf902a46ddb2b74c5f79e510f2; no voters:
I20260928 05:23:44.212810 16782 raft_consensus.cc:2781] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:44.212946 16782 raft_consensus.cc:482] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:44.213047 16782 raft_consensus.cc:3037] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.215081 16782 raft_consensus.cc:504] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.215623 16782 leader_election.cc:288] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Requested vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.215976 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14836daf25d7423b90532a1f6b2f7cf8" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8"
I20260928 05:23:44.216012 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14836daf25d7423b90532a1f6b2f7cf8" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2"
I20260928 05:23:44.216146 16561 raft_consensus.cc:3037] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.216156 16705 raft_consensus.cc:3037] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.218253 16705 raft_consensus.cc:2454] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:44.218278 16561 raft_consensus.cc:2454] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:44.218585 16352 leader_election.cc:302] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, df8dd992665445f2b51ab503c874cea0; no voters:
I20260928 05:23:44.218837 16782 raft_consensus.cc:2781] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:44.219228 16782 raft_consensus.cc:686] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Becoming Leader. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Running, Role: LEADER
I20260928 05:23:44.219833 16782 consensus_queue.cc:237] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.223882 16265 catalog_manager.cc:5495] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8dd992665445f2b51ab503c874cea0 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "df8dd992665445f2b51ab503c874cea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:44.250029 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.245147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.8.124.193:45039)
0928 05:23:44.245351 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.245356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.245460 (+ 104us) server_negotiation.cc:407] Connection header received
0928 05:23:44.245674 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.245678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.245764 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.245920 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:44.246262 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.246969 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.248199 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.248562 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.248643 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.249026 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.249526 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.249657 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.249728 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.249786 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:23:44.257483 16705 raft_consensus.cc:1261] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.257556 16561 raft_consensus.cc:1261] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.258343 16782 consensus_queue.cc:1035] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.258603 16786 consensus_queue.cc:1035] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.267360 16789 mvcc.cc:205] Tried to move back new op lower bound from 7334187107350052864 to 7334187107207585792. Current Snapshot: MvccSnapshot[applied={T|T < 7334187107350052864}]
I20260928 05:23:44.268229 16791 mvcc.cc:205] Tried to move back new op lower bound from 7334187107350052864 to 7334187107207585792. Current Snapshot: MvccSnapshot[applied={T|T < 7334187107350052864}]
I20260928 05:23:44.283224 16265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:44.283521 16265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:44.286749 16265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Sending DeleteTablet for 3 replicas of tablet 14836daf25d7423b90532a1f6b2f7cf8
I20260928 05:23:44.287848 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60398
I20260928 05:23:44.288159 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39472
I20260928 05:23:44.288769 16799 tablet_replica.cc:330] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: stopping tablet replica
I20260928 05:23:44.288793 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet 14836daf25d7423b90532a1f6b2f7cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:48432
I20260928 05:23:44.288841 16264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:23:44.289175 16799 raft_consensus.cc:2229] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Raft consensus shutting down.
W20260928 05:23:44.289577 16264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:44.289834 16799 raft_consensus.cc:2258] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.291831 16799 ts_tablet_manager.cc:1905] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.298185 16684 tablet_service.cc:1467] Processing CreateTablet for tablet d8e245edf39a47a88fff59d8fa692f7b (DEFAULT_TABLE table=test-table [id=8a46cb53b05f4cc38d918d10d134a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.298705 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e245edf39a47a88fff59d8fa692f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.298827 16799 ts_tablet_manager.cc:1918] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:44.298977 16799 log.cc:1196] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/14836daf25d7423b90532a1f6b2f7cf8
I20260928 05:23:44.299461 16799 ts_tablet_manager.cc:1939] T 14836daf25d7423b90532a1f6b2f7cf8 P df8dd992665445f2b51ab503c874cea0: Deleting consensus metadata
I20260928 05:23:44.302126 16251 catalog_manager.cc:4841] TS df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039): tablet 14836daf25d7423b90532a1f6b2f7cf8 (table test-table [id=fa716efd04f744b699dbd9e2a301650e]) successfully deleted
I20260928 05:23:44.302474 16800 tablet_replica.cc:330] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: stopping tablet replica
I20260928 05:23:44.302363 16540 tablet_service.cc:1467] Processing CreateTablet for tablet d8e245edf39a47a88fff59d8fa692f7b (DEFAULT_TABLE table=test-table [id=8a46cb53b05f4cc38d918d10d134a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.302915 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e245edf39a47a88fff59d8fa692f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.302937 16800 raft_consensus.cc:2229] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:44.303537 16800 raft_consensus.cc:2258] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.305967 16777 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Bootstrap starting.
I20260928 05:23:44.306121 16401 tablet_service.cc:1467] Processing CreateTablet for tablet d8e245edf39a47a88fff59d8fa692f7b (DEFAULT_TABLE table=test-table [id=8a46cb53b05f4cc38d918d10d134a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:44.306692 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e245edf39a47a88fff59d8fa692f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.307358 16777 tablet_bootstrap.cc:654] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.309243 16777 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: No bootstrap required, opened a new log
I20260928 05:23:44.309337 16777 ts_tablet_manager.cc:1397] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 05:23:44.310572 16777 raft_consensus.cc:348] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.310786 16777 raft_consensus.cc:374] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.310879 16777 raft_consensus.cc:729] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.311177 16777 consensus_queue.cc:260] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.313419 16776 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Bootstrap starting.
I20260928 05:23:44.313418 16775 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Bootstrap starting.
I20260928 05:23:44.314074 16801 tablet_replica.cc:330] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: stopping tablet replica
I20260928 05:23:44.314525 16801 raft_consensus.cc:2229] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:44.314967 16776 tablet_bootstrap.cc:654] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.314966 16775 tablet_bootstrap.cc:654] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.315073 16801 raft_consensus.cc:2258] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.315272 16777 ts_tablet_manager.cc:1428] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 05:23:44.317286 16800 ts_tablet_manager.cc:1905] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.320113 16801 ts_tablet_manager.cc:1905] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.323971 16800 ts_tablet_manager.cc:1918] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:44.324204 16800 log.cc:1196] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/14836daf25d7423b90532a1f6b2f7cf8
I20260928 05:23:44.324765 16800 ts_tablet_manager.cc:1939] T 14836daf25d7423b90532a1f6b2f7cf8 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting consensus metadata
I20260928 05:23:44.325649 16775 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: No bootstrap required, opened a new log
I20260928 05:23:44.325843 16775 ts_tablet_manager.cc:1397] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260928 05:23:44.326799 16249 catalog_manager.cc:4841] TS 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371): tablet 14836daf25d7423b90532a1f6b2f7cf8 (table test-table [id=fa716efd04f744b699dbd9e2a301650e]) successfully deleted
I20260928 05:23:44.326711 16775 raft_consensus.cc:348] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.326958 16775 raft_consensus.cc:374] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.327049 16775 raft_consensus.cc:729] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.327310 16775 consensus_queue.cc:260] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.328008 16775 ts_tablet_manager.cc:1428] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:23:44.329372 16776 tablet_bootstrap.cc:492] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: No bootstrap required, opened a new log
I20260928 05:23:44.329507 16776 ts_tablet_manager.cc:1397] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20260928 05:23:44.330297 16776 raft_consensus.cc:348] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.330528 16776 raft_consensus.cc:374] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.330608 16776 raft_consensus.cc:729] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.330904 16776 consensus_queue.cc:260] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.331943 16776 ts_tablet_manager.cc:1428] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:44.333964 16801 ts_tablet_manager.cc:1918] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:44.334115 16801 log.cc:1196] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/14836daf25d7423b90532a1f6b2f7cf8
I20260928 05:23:44.334579 16801 ts_tablet_manager.cc:1939] T 14836daf25d7423b90532a1f6b2f7cf8 P e19018bf902a46ddb2b74c5f79e510f2: Deleting consensus metadata
I20260928 05:23:44.335955 16249 catalog_manager.cc:4841] TS e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043): tablet 14836daf25d7423b90532a1f6b2f7cf8 (table test-table [id=fa716efd04f744b699dbd9e2a301650e]) successfully deleted
I20260928 05:23:44.448076 16796 raft_consensus.cc:482] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:44.448318 16796 raft_consensus.cc:504] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.449182 16796 leader_election.cc:288] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.449798 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e245edf39a47a88fff59d8fa692f7b" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2" is_pre_election: true
I20260928 05:23:44.449826 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e245edf39a47a88fff59d8fa692f7b" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8" is_pre_election: true
I20260928 05:23:44.450023 16705 raft_consensus.cc:2454] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:44.450069 16561 raft_consensus.cc:2454] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:44.450492 16352 leader_election.cc:302] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, df8dd992665445f2b51ab503c874cea0; no voters:
I20260928 05:23:44.450752 16796 raft_consensus.cc:2781] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:44.450886 16796 raft_consensus.cc:482] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:44.450968 16796 raft_consensus.cc:3037] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.454098 16796 raft_consensus.cc:504] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.454710 16796 leader_election.cc:288] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Requested vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.455238 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e245edf39a47a88fff59d8fa692f7b" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2"
I20260928 05:23:44.455289 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e245edf39a47a88fff59d8fa692f7b" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8"
I20260928 05:23:44.455438 16705 raft_consensus.cc:3037] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.455492 16561 raft_consensus.cc:3037] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.457526 16705 raft_consensus.cc:2454] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:44.457916 16352 leader_election.cc:302] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df8dd992665445f2b51ab503c874cea0, e19018bf902a46ddb2b74c5f79e510f2; no voters:
I20260928 05:23:44.458180 16796 raft_consensus.cc:2781] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:44.458325 16796 raft_consensus.cc:686] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Becoming Leader. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Running, Role: LEADER
I20260928 05:23:44.458385 16561 raft_consensus.cc:2454] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:44.458611 16796 consensus_queue.cc:237] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.461310 16264 catalog_manager.cc:5495] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8dd992665445f2b51ab503c874cea0 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "df8dd992665445f2b51ab503c874cea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:23:44 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573021, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.254@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:44.516316 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.499690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.8.124.254:37625)
0928 05:23:44.500564 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.500568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.500586 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:23:44.500622 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.500625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.500673 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.500755 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:44.501878 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.502665 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.503826 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.504218 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.504318 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.511178 (+ 6860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:44.511183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:44.511186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:44.511213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:44.513504 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:44.514338 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:44.514342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:44.514344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:44.514445 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:44.514809 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:44.514813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:44.514815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:44.515067 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:44.515165 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.515666 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.516019 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":767,"thread_start_us":626,"threads_started":1}
I20260928 05:23:44.516376 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.499632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37625 (local address 127.0.0.1:49976)
0928 05:23:44.500251 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.500270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.500422 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.500802 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.500807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.501457 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:44.501736 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.501748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.502779 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.502785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.503719 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.503731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.504375 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.504382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.504535 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.508477 (+ 3942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:44.508494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:44.510990 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:44.513664 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:44.513677 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:44.513685 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:44.514235 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:44.514538 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:44.514543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:44.514547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:44.514709 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:44.515187 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:44.515196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:44.515402 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.515575 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.515895 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260928 05:23:44.520925 16264 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:49976:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:23:44.521926 16264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:44.528350 16401 tablet_service.cc:1467] Processing CreateTablet for tablet a86c8ea2d0224bd79453c81c70133aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e32f7cb53f24db5b019384d7d8ebbd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:44.528643 16541 tablet_service.cc:1467] Processing CreateTablet for tablet a86c8ea2d0224bd79453c81c70133aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e32f7cb53f24db5b019384d7d8ebbd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:44.528888 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86c8ea2d0224bd79453c81c70133aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.529105 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86c8ea2d0224bd79453c81c70133aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.533972 16776 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: Bootstrap starting.
I20260928 05:23:44.534102 16685 tablet_service.cc:1467] Processing CreateTablet for tablet a86c8ea2d0224bd79453c81c70133aef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e32f7cb53f24db5b019384d7d8ebbd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:23:44.534605 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86c8ea2d0224bd79453c81c70133aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:44.535511 16776 tablet_bootstrap.cc:654] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.535950 16775 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: Bootstrap starting.
I20260928 05:23:44.537557 16775 tablet_bootstrap.cc:654] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.542136 16777 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: Bootstrap starting.
I20260928 05:23:44.543777 16777 tablet_bootstrap.cc:654] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:44.548039 16776 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: No bootstrap required, opened a new log
I20260928 05:23:44.548219 16776 ts_tablet_manager.cc:1397] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260928 05:23:44.548971 16776 raft_consensus.cc:348] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.549234 16776 raft_consensus.cc:374] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.549309 16776 raft_consensus.cc:729] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.549590 16776 consensus_queue.cc:260] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.550045 16783 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.550417 16783 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.553177 16776 ts_tablet_manager.cc:1428] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260928 05:23:44.553594 16777 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: No bootstrap required, opened a new log
I20260928 05:23:44.553836 16777 ts_tablet_manager.cc:1397] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260928 05:23:44.554584 16777 raft_consensus.cc:348] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.554876 16777 raft_consensus.cc:374] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.554996 16777 raft_consensus.cc:729] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.555302 16777 consensus_queue.cc:260] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.555781 16781 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.556078 16781 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.556828 16777 ts_tablet_manager.cc:1428] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 05:23:44.558506 16775 tablet_bootstrap.cc:492] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: No bootstrap required, opened a new log
I20260928 05:23:44.558686 16775 ts_tablet_manager.cc:1397] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20260928 05:23:44.559535 16775 raft_consensus.cc:348] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.559787 16775 raft_consensus.cc:374] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:44.559892 16775 raft_consensus.cc:729] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Initialized, Role: FOLLOWER
I20260928 05:23:44.560184 16775 consensus_queue.cc:260] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.560637 16796 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.560940 16796 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.561429 16775 ts_tablet_manager.cc:1428] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:44.747259 16783 raft_consensus.cc:482] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:44.747489 16783 raft_consensus.cc:504] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.749019 16783 leader_election.cc:288] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.755415 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.749299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:41043 (local address 127.8.124.194:45809)
0928 05:23:44.749901 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.749923 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.750063 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.750456 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.750462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.750490 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.750778 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.750791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.751968 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.751974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.753476 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.753488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.754673 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.754678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.754847 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.754882 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.754966 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.755060 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
I20260928 05:23:44.755465 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.749496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:45809 (local address 127.8.124.195:41043)
0928 05:23:44.750118 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.750124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.750146 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:23:44.750220 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.750225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.750299 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.750421 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.750960 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.751826 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.753677 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.754516 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.754642 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.754691 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.755012 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.755139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":418,"threads_started":1}
I20260928 05:23:44.755543 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.749307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:41347 (local address 127.8.124.193:45039)
0928 05:23:44.749499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.749505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.750483 (+ 978us) server_negotiation.cc:407] Connection header received
0928 05:23:44.750634 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.750639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.750709 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.750801 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:44.751386 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.752161 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.753926 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.754683 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.754825 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.754889 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.755189 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.755315 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:23:44.755669 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.749303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:45039 (local address 127.8.124.194:41347)
0928 05:23:44.750394 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.750410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.750517 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.750812 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.750815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.750836 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:44.751216 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.751227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.752307 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.752312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.753784 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.753797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.754877 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.754882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.755033 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.755064 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.755157 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.755276 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":51,"thread_start_us":864,"threads_started":1}
I20260928 05:23:44.756412 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86c8ea2d0224bd79453c81c70133aef" candidate_uuid: "214b60f54ab44fdab111bb78c4ab87a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2" is_pre_election: true
I20260928 05:23:44.756656 16705 raft_consensus.cc:2454] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214b60f54ab44fdab111bb78c4ab87a8 in term 0.
I20260928 05:23:44.756657 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86c8ea2d0224bd79453c81c70133aef" candidate_uuid: "214b60f54ab44fdab111bb78c4ab87a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dd992665445f2b51ab503c874cea0" is_pre_election: true
I20260928 05:23:44.756983 16426 raft_consensus.cc:2454] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214b60f54ab44fdab111bb78c4ab87a8 in term 0.
I20260928 05:23:44.757042 16492 leader_election.cc:302] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, e19018bf902a46ddb2b74c5f79e510f2; no voters:
I20260928 05:23:44.757409 16783 raft_consensus.cc:2781] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:44.757512 16783 raft_consensus.cc:482] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:44.757627 16783 raft_consensus.cc:3037] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.759637 16783 raft_consensus.cc:504] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.760260 16783 leader_election.cc:288] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [CANDIDATE]: Term 1 election: Requested vote from peers df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:44.760684 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86c8ea2d0224bd79453c81c70133aef" candidate_uuid: "214b60f54ab44fdab111bb78c4ab87a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dd992665445f2b51ab503c874cea0"
I20260928 05:23:44.760694 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86c8ea2d0224bd79453c81c70133aef" candidate_uuid: "214b60f54ab44fdab111bb78c4ab87a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2"
I20260928 05:23:44.760864 16426 raft_consensus.cc:3037] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.760903 16705 raft_consensus.cc:3037] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:44.762868 16426 raft_consensus.cc:2454] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214b60f54ab44fdab111bb78c4ab87a8 in term 1.
I20260928 05:23:44.762898 16705 raft_consensus.cc:2454] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214b60f54ab44fdab111bb78c4ab87a8 in term 1.
I20260928 05:23:44.763219 16494 leader_election.cc:302] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, df8dd992665445f2b51ab503c874cea0; no voters:
I20260928 05:23:44.763477 16783 raft_consensus.cc:2781] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:44.763859 16783 raft_consensus.cc:686] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 LEADER]: Becoming Leader. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Running, Role: LEADER
I20260928 05:23:44.764230 16783 consensus_queue.cc:237] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:44.765983 16819 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: TxnStatusTablet state changed. Reason: New leader 214b60f54ab44fdab111bb78c4ab87a8. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.766347 16819 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:44.766980 16821 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:23:44.767184 16821 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:23:44.767797 16265 catalog_manager.cc:5495] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:44.789999 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.783549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.8.124.194:44371)
0928 05:23:44.784077 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.784082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.784097 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:44.784167 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.784170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.784240 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.784378 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:44.784774 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.785805 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.787636 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.788104 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.788196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.788738 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.789315 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.789453 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.789632 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.789721 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
I20260928 05:23:44.790050 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.783534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44371 (local address 127.0.0.1:39488)
0928 05:23:44.783753 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.783773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.783911 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.784408 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.784413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.784434 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:44.784623 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.784633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.785929 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.785934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.787494 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.787509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.788242 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.788248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.788461 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.788530 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.789440 (+ 910us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.789462 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.789549 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.789734 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:23:44.794818 16705 raft_consensus.cc:1261] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Refusing update from remote peer 214b60f54ab44fdab111bb78c4ab87a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.794937 16426 raft_consensus.cc:1261] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Refusing update from remote peer 214b60f54ab44fdab111bb78c4ab87a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.795601 16783 consensus_queue.cc:1035] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.795866 16819 consensus_queue.cc:1035] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.802637 16781 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: TxnStatusTablet state changed. Reason: New leader 214b60f54ab44fdab111bb78c4ab87a8. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.802924 16781 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.805241 16796 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: TxnStatusTablet state changed. Reason: New leader 214b60f54ab44fdab111bb78c4ab87a8. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.805495 16796 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.805832 16826 mvcc.cc:205] Tried to move back new op lower bound from 7334187109552353280 to 7334187109437607936. Current Snapshot: MvccSnapshot[applied={T|T < 7334187109552353280}]
I20260928 05:23:44.806684 16819 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.806921 16819 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:44.807401 16819 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.812846 16819 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P 214b60f54ab44fdab111bb78c4ab87a8: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:23:44.813544 16781 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.813797 16781 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P e19018bf902a46ddb2b74c5f79e510f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.814682 16796 tablet_replica.cc:439] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "214b60f54ab44fdab111bb78c4ab87a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } } }
I20260928 05:23:44.814939 16796 tablet_replica.cc:442] T a86c8ea2d0224bd79453c81c70133aef P df8dd992665445f2b51ab503c874cea0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:23:44.834200 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.828806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:49847 (local address 127.8.124.194:44371)
0928 05:23:44.828943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.828949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.829311 (+ 362us) server_negotiation.cc:407] Connection header received
0928 05:23:44.829492 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.829495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.829555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.829660 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:44.830047 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.830805 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.832372 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.833063 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.833189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.833528 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.833676 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.833764 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.833855 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.833932 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:23:44.834220 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.828773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44371 (local address 127.8.124.193:49847)
0928 05:23:44.829218 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.829234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.829359 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.829670 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.829675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.829686 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:44.829915 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.829931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.830938 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.830942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.832207 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.832223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.833205 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.833209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.833317 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.833367 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.833760 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.833767 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.833837 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.833965 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":227,"threads_started":1}
I20260928 05:23:44.843469 16705 raft_consensus.cc:1261] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.843590 16561 raft_consensus.cc:1261] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:44.844067 16796 consensus_queue.cc:1035] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.844377 16782 consensus_queue.cc:1035] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:44.879115 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.870128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:58593 (local address 127.8.124.193:45039)
0928 05:23:44.870297 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.870303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.873104 (+ 2801us) server_negotiation.cc:407] Connection header received
0928 05:23:44.873272 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.873277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.873350 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.873425 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:44.874014 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.875066 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.877209 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.877640 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.877743 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.878419 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.878539 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.878651 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:44.878773 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 05:23:44.878851 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:23:44.879312 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:44.870184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:45039 (local address 127.8.124.194:58593)
0928 05:23:44.873009 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:44.873024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:44.873152 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:44.873455 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:44.873459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:44.873507 (+ 48us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:23:44.873851 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.873875 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.875227 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.875233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.877074 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:44.877089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.877820 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.877827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:44.878011 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.878078 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:23:44.878659 (+ 581us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:23:44.878671 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 05:23:44.878752 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 05:23:44.878891 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2639,"thread_start_us":2446,"threads_started":1}
I20260928 05:23:44.914827 16265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:44.915037 16265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:44.917898 16265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Sending DeleteTablet for 3 replicas of tablet d8e245edf39a47a88fff59d8fa692f7b
I20260928 05:23:44.918612 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e245edf39a47a88fff59d8fa692f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60398
I20260928 05:23:44.918814 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e245edf39a47a88fff59d8fa692f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:48432
I20260928 05:23:44.919400 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e245edf39a47a88fff59d8fa692f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:44 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39472
I20260928 05:23:44.928632 16844 tablet_replica.cc:330] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: stopping tablet replica
I20260928 05:23:44.928853 16844 raft_consensus.cc:2229] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:44.929160 16844 raft_consensus.cc:2258] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.930882 16842 tablet_replica.cc:330] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: stopping tablet replica
I20260928 05:23:44.931149 16842 raft_consensus.cc:2229] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:44.931603 16842 raft_consensus.cc:2258] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.931710 16844 ts_tablet_manager.cc:1905] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.937585 16843 tablet_replica.cc:330] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: stopping tablet replica
I20260928 05:23:44.937826 16843 raft_consensus.cc:2229] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:44.938071 16843 raft_consensus.cc:2258] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:44.939165 16843 ts_tablet_manager.cc:1905] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.939348 16842 ts_tablet_manager.cc:1905] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:44.947777 16843 ts_tablet_manager.cc:1918] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:44.947922 16843 log.cc:1196] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/d8e245edf39a47a88fff59d8fa692f7b
I20260928 05:23:44.948383 16843 ts_tablet_manager.cc:1939] T d8e245edf39a47a88fff59d8fa692f7b P e19018bf902a46ddb2b74c5f79e510f2: Deleting consensus metadata
I20260928 05:23:44.949556 16249 catalog_manager.cc:4841] TS e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043): tablet d8e245edf39a47a88fff59d8fa692f7b (table test-table [id=8a46cb53b05f4cc38d918d10d134a176]) successfully deleted
I20260928 05:23:44.949882 16844 ts_tablet_manager.cc:1918] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:44.950023 16844 log.cc:1196] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/d8e245edf39a47a88fff59d8fa692f7b
I20260928 05:23:44.950479 16844 ts_tablet_manager.cc:1939] T d8e245edf39a47a88fff59d8fa692f7b P 214b60f54ab44fdab111bb78c4ab87a8: Deleting consensus metadata
I20260928 05:23:44.950799 16842 ts_tablet_manager.cc:1918] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:44.950927 16842 log.cc:1196] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/d8e245edf39a47a88fff59d8fa692f7b
I20260928 05:23:44.951381 16842 ts_tablet_manager.cc:1939] T d8e245edf39a47a88fff59d8fa692f7b P df8dd992665445f2b51ab503c874cea0: Deleting consensus metadata
I20260928 05:23:44.951642 16249 catalog_manager.cc:4841] TS 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371): tablet d8e245edf39a47a88fff59d8fa692f7b (table test-table [id=8a46cb53b05f4cc38d918d10d134a176]) successfully deleted
I20260928 05:23:44.952412 16251 catalog_manager.cc:4841] TS df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039): tablet d8e245edf39a47a88fff59d8fa692f7b (table test-table [id=8a46cb53b05f4cc38d918d10d134a176]) successfully deleted
Sep 28 05:23:44 dist-test-slave-515x krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:23:44 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:44 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:23:44.998752 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:44.988540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.8.124.254:37625)
0928 05:23:44.988759 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:44.988766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:44.989271 (+ 505us) server_negotiation.cc:407] Connection header received
0928 05:23:44.989457 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:44.989461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:44.989527 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:44.989626 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:44.990851 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.991637 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.992947 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:44.993379 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:44.993487 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:44.998491 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49982: BlockingRecv error: recv got EOF from 127.0.0.1:49982 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 05:23:45 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM
I20260928 05:23:45.015264 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.001411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.8.124.254:37625)
0928 05:23:45.001586 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.001592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.002010 (+ 418us) server_negotiation.cc:407] Connection header received
0928 05:23:45.002156 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.002162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.002239 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.002350 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:45.003453 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.004537 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.005757 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.006406 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.006526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.010355 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:45.010360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:45.010364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:45.010399 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:45.012803 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.013556 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.013559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.013562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.013718 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.014107 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.014110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.014112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.014357 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:45.014536 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.014835 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.015006 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:23:45.018676 16265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:23:45.019271 16265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:45.025301 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 (DEFAULT_TABLE table=test-table [id=f038ebcf54e4486d85f80a6456a50c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.025866 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb665bc195444ba3afcb32415fbc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.026100 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 (DEFAULT_TABLE table=test-table [id=f038ebcf54e4486d85f80a6456a50c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.026468 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb665bc195444ba3afcb32415fbc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.031524 16776 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Bootstrap starting.
I20260928 05:23:45.031893 16685 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 (DEFAULT_TABLE table=test-table [id=f038ebcf54e4486d85f80a6456a50c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.032289 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb665bc195444ba3afcb32415fbc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.033193 16776 tablet_bootstrap.cc:654] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.037830 16775 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Bootstrap starting.
I20260928 05:23:45.039350 16775 tablet_bootstrap.cc:654] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.039383 16776 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: No bootstrap required, opened a new log
I20260928 05:23:45.039515 16776 ts_tablet_manager.cc:1397] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:23:45.039772 16777 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Bootstrap starting.
I20260928 05:23:45.040246 16776 raft_consensus.cc:348] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.040588 16776 raft_consensus.cc:374] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.040750 16776 raft_consensus.cc:729] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.041101 16777 tablet_bootstrap.cc:654] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.041085 16776 consensus_queue.cc:260] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.044409 16776 ts_tablet_manager.cc:1428] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:23:45.049896 16777 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: No bootstrap required, opened a new log
I20260928 05:23:45.049958 16775 tablet_bootstrap.cc:492] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: No bootstrap required, opened a new log
I20260928 05:23:45.050041 16777 ts_tablet_manager.cc:1397] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 05:23:45.050144 16775 ts_tablet_manager.cc:1397] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260928 05:23:45.050673 16777 raft_consensus.cc:348] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.050863 16777 raft_consensus.cc:374] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.050974 16777 raft_consensus.cc:729] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.050915 16775 raft_consensus.cc:348] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.051189 16775 raft_consensus.cc:374] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.051294 16775 raft_consensus.cc:729] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.051189 16777 consensus_queue.cc:260] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.051512 16775 consensus_queue.cc:260] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.052073 16777 ts_tablet_manager.cc:1428] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:45.053104 16775 ts_tablet_manager.cc:1428] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:45.193161 16781 raft_consensus.cc:482] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:45.193396 16781 raft_consensus.cc:504] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.195108 16781 leader_election.cc:288] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039), 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371)
I20260928 05:23:45.203984 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.196143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:45039 (local address 127.8.124.195:59681)
0928 05:23:45.196692 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:45.196711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:45.196864 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:45.197220 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:45.197225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:45.197254 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:45.197721 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:45.197734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.198912 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.198919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:45.201379 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:45.201397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.203057 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.203063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:45.203258 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.203297 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 05:23:45.203390 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:23:45.203535 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":675,"threads_started":1}
I20260928 05:23:45.204123 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.195532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:59681 (local address 127.8.124.193:45039)
0928 05:23:45.195824 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.195831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.196814 (+ 983us) server_negotiation.cc:407] Connection header received
0928 05:23:45.197015 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.197020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.197095 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.197192 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:45.197881 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.198775 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.201548 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.202814 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.202983 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.203057 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.203433 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.203564 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260928 05:23:45.205024 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb665bc195444ba3afcb32415fbc27" candidate_uuid: "e19018bf902a46ddb2b74c5f79e510f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dd992665445f2b51ab503c874cea0" is_pre_election: true
I20260928 05:23:45.205228 16426 raft_consensus.cc:2454] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e19018bf902a46ddb2b74c5f79e510f2 in term 0.
I20260928 05:23:45.205713 16633 leader_election.cc:302] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df8dd992665445f2b51ab503c874cea0, e19018bf902a46ddb2b74c5f79e510f2; no voters:
I20260928 05:23:45.206130 16781 raft_consensus.cc:2781] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:45.206246 16781 raft_consensus.cc:482] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:45.206332 16781 raft_consensus.cc:3037] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.209275 16781 raft_consensus.cc:504] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.209838 16781 leader_election.cc:288] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [CANDIDATE]: Term 1 election: Requested vote from peers df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039), 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371)
I20260928 05:23:45.210265 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb665bc195444ba3afcb32415fbc27" candidate_uuid: "e19018bf902a46ddb2b74c5f79e510f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dd992665445f2b51ab503c874cea0"
I20260928 05:23:45.210467 16426 raft_consensus.cc:3037] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.210759 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.196585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44371 (local address 127.8.124.195:59421)
0928 05:23:45.204707 (+ 8122us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:45.204725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:45.204913 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:45.205312 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:45.205317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:45.205343 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:23:45.205502 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:45.205514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.206762 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.206767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:45.208661 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:45.208675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.210073 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.210079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:45.210258 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.210281 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:23:45.210378 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:23:45.210517 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8018,"thread_start_us":8232,"threads_started":1}
I20260928 05:23:45.210824 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.195596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:59421 (local address 127.8.124.194:44371)
0928 05:23:45.195785 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.195792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.204796 (+ 9004us) server_negotiation.cc:407] Connection header received
0928 05:23:45.205093 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.205098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.205169 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.205266 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:23:45.205670 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.206607 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.208803 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.209755 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.209899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.209969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.210426 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.210574 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:23:45.211581 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb665bc195444ba3afcb32415fbc27" candidate_uuid: "e19018bf902a46ddb2b74c5f79e510f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8" is_pre_election: true
I20260928 05:23:45.211719 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb665bc195444ba3afcb32415fbc27" candidate_uuid: "e19018bf902a46ddb2b74c5f79e510f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8"
I20260928 05:23:45.211838 16561 raft_consensus.cc:2454] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e19018bf902a46ddb2b74c5f79e510f2 in term 0.
I20260928 05:23:45.212855 16426 raft_consensus.cc:2454] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e19018bf902a46ddb2b74c5f79e510f2 in term 1.
I20260928 05:23:45.213181 16633 leader_election.cc:302] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df8dd992665445f2b51ab503c874cea0, e19018bf902a46ddb2b74c5f79e510f2; no voters: 214b60f54ab44fdab111bb78c4ab87a8
I20260928 05:23:45.213495 16781 raft_consensus.cc:2781] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:45.213881 16781 raft_consensus.cc:686] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 LEADER]: Becoming Leader. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Running, Role: LEADER
I20260928 05:23:45.214203 16781 consensus_queue.cc:237] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.217254 16264 catalog_manager.cc:5495] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195). New cstate: current_term: 1 leader_uuid: "e19018bf902a46ddb2b74c5f79e510f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } health_report { overall_health: HEALTHY } } }
I20260928 05:23:45.233624 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.228785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.8.124.195:41043)
0928 05:23:45.228999 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.229004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.229073 (+ 69us) server_negotiation.cc:407] Connection header received
0928 05:23:45.229233 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.229237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.229316 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.229438 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:45.229796 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.230475 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.231669 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.232085 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.232167 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.232538 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:45.233085 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:45.233183 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.233273 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.233332 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:23:45.236621 16560 raft_consensus.cc:3037] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.236778 16426 raft_consensus.cc:1261] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Refusing update from remote peer e19018bf902a46ddb2b74c5f79e510f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:45.237321 16862 consensus_queue.cc:1035] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:45.238960 16560 raft_consensus.cc:1261] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Refusing update from remote peer e19018bf902a46ddb2b74c5f79e510f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:45.239506 16862 consensus_queue.cc:1035] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:45.251948 16264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:45.252172 16264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:45.254910 16264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Sending DeleteTablet for 3 replicas of tablet 8fcb665bc195444ba3afcb32415fbc27
I20260928 05:23:45.255574 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60398
I20260928 05:23:45.255717 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39472
I20260928 05:23:45.255849 16842 tablet_replica.cc:330] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: stopping tablet replica
I20260928 05:23:45.255940 16844 tablet_replica.cc:330] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: stopping tablet replica
I20260928 05:23:45.256042 16842 raft_consensus.cc:2229] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:45.256122 16844 raft_consensus.cc:2229] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:45.256246 16842 raft_consensus.cc:2258] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.256316 16844 raft_consensus.cc:2258] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.256630 16264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:23:45.257228 16844 ts_tablet_manager.cc:1905] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 05:23:45.257223 16264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:23:45.257354 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcb665bc195444ba3afcb32415fbc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:48432
I20260928 05:23:45.257583 16843 tablet_replica.cc:330] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: stopping tablet replica
I20260928 05:23:45.257777 16843 raft_consensus.cc:2229] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:45.258236 16843 raft_consensus.cc:2258] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.258406 16842 ts_tablet_manager.cc:1905] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:45.259554 16843 ts_tablet_manager.cc:1905] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:45.263259 16844 ts_tablet_manager.cc:1918] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:45.263404 16844 log.cc:1196] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/8fcb665bc195444ba3afcb32415fbc27
I20260928 05:23:45.263806 16844 ts_tablet_manager.cc:1939] T 8fcb665bc195444ba3afcb32415fbc27 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting consensus metadata
I20260928 05:23:45.264446 16541 tablet_service.cc:1467] Processing CreateTablet for tablet ecedba6ec315460f8aa88683fe1085c5 (DEFAULT_TABLE table=test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.264809 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecedba6ec315460f8aa88683fe1085c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.264791 16401 tablet_service.cc:1467] Processing CreateTablet for tablet ecedba6ec315460f8aa88683fe1085c5 (DEFAULT_TABLE table=test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.264995 16842 ts_tablet_manager.cc:1918] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:45.265149 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecedba6ec315460f8aa88683fe1085c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.265293 16842 log.cc:1196] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/8fcb665bc195444ba3afcb32415fbc27
I20260928 05:23:45.265589 16843 ts_tablet_manager.cc:1918] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:23:45.265758 16843 log.cc:1196] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/8fcb665bc195444ba3afcb32415fbc27
I20260928 05:23:45.265779 16842 ts_tablet_manager.cc:1939] T 8fcb665bc195444ba3afcb32415fbc27 P df8dd992665445f2b51ab503c874cea0: Deleting consensus metadata
I20260928 05:23:45.266144 16685 tablet_service.cc:1467] Processing CreateTablet for tablet ecedba6ec315460f8aa88683fe1085c5 (DEFAULT_TABLE table=test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:23:45.266525 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecedba6ec315460f8aa88683fe1085c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:45.266156 16843 ts_tablet_manager.cc:1939] T 8fcb665bc195444ba3afcb32415fbc27 P e19018bf902a46ddb2b74c5f79e510f2: Deleting consensus metadata
I20260928 05:23:45.270308 16249 catalog_manager.cc:4841] TS e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043): tablet 8fcb665bc195444ba3afcb32415fbc27 (table test-table [id=f038ebcf54e4486d85f80a6456a50c6e]) successfully deleted
I20260928 05:23:45.270480 16776 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Bootstrap starting.
I20260928 05:23:45.270606 16251 catalog_manager.cc:4841] TS df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039): tablet 8fcb665bc195444ba3afcb32415fbc27 (table test-table [id=f038ebcf54e4486d85f80a6456a50c6e]) successfully deleted
I20260928 05:23:45.271919 16776 tablet_bootstrap.cc:654] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.273710 16775 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Bootstrap starting.
I20260928 05:23:45.273761 16776 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: No bootstrap required, opened a new log
I20260928 05:23:45.273898 16776 ts_tablet_manager.cc:1397] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 05:23:45.274266 16777 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Bootstrap starting.
I20260928 05:23:45.275218 16775 tablet_bootstrap.cc:654] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.275650 16777 tablet_bootstrap.cc:654] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:45.276862 16776 raft_consensus.cc:348] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.277122 16776 raft_consensus.cc:374] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.277195 16776 raft_consensus.cc:729] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b60f54ab44fdab111bb78c4ab87a8, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.277422 16776 consensus_queue.cc:260] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.278391 16775 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: No bootstrap required, opened a new log
I20260928 05:23:45.278600 16775 ts_tablet_manager.cc:1397] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 05:23:45.280325 16776 ts_tablet_manager.cc:1428] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260928 05:23:45.280368 16775 raft_consensus.cc:348] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.280731 16775 raft_consensus.cc:374] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.280836 16775 raft_consensus.cc:729] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.281141 16775 consensus_queue.cc:260] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.283010 16249 catalog_manager.cc:4841] TS 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371): tablet 8fcb665bc195444ba3afcb32415fbc27 (table test-table [id=f038ebcf54e4486d85f80a6456a50c6e]) successfully deleted
I20260928 05:23:45.283981 16775 ts_tablet_manager.cc:1428] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:23:45.286056 16777 tablet_bootstrap.cc:492] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: No bootstrap required, opened a new log
I20260928 05:23:45.286211 16777 ts_tablet_manager.cc:1397] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260928 05:23:45.286857 16777 raft_consensus.cc:348] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.287075 16777 raft_consensus.cc:374] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:45.287144 16777 raft_consensus.cc:729] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19018bf902a46ddb2b74c5f79e510f2, State: Initialized, Role: FOLLOWER
I20260928 05:23:45.287374 16777 consensus_queue.cc:260] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.288414 16777 ts_tablet_manager.cc:1428] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:23:45.396631 16796 raft_consensus.cc:482] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:23:45.396828 16796 raft_consensus.cc:504] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.397490 16796 leader_election.cc:288] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:45.398052 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecedba6ec315460f8aa88683fe1085c5" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8" is_pre_election: true
I20260928 05:23:45.398300 16560 raft_consensus.cc:2454] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:45.398317 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecedba6ec315460f8aa88683fe1085c5" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2" is_pre_election: true
I20260928 05:23:45.398598 16705 raft_consensus.cc:2454] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 0.
I20260928 05:23:45.398747 16352 leader_election.cc:302] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, df8dd992665445f2b51ab503c874cea0; no voters:
I20260928 05:23:45.399159 16796 raft_consensus.cc:2781] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:23:45.399259 16796 raft_consensus.cc:482] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:23:45.399343 16796 raft_consensus.cc:3037] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.401254 16796 raft_consensus.cc:504] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.401782 16796 leader_election.cc:288] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Requested vote from peers 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371), e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043)
I20260928 05:23:45.402246 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecedba6ec315460f8aa88683fe1085c5" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b60f54ab44fdab111bb78c4ab87a8"
I20260928 05:23:45.402457 16560 raft_consensus.cc:3037] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.402491 16705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecedba6ec315460f8aa88683fe1085c5" candidate_uuid: "df8dd992665445f2b51ab503c874cea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19018bf902a46ddb2b74c5f79e510f2"
I20260928 05:23:45.402688 16705 raft_consensus.cc:3037] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:45.404500 16560 raft_consensus.cc:2454] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:45.404850 16352 leader_election.cc:302] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214b60f54ab44fdab111bb78c4ab87a8, df8dd992665445f2b51ab503c874cea0; no voters:
I20260928 05:23:45.405138 16796 raft_consensus.cc:2781] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:45.405500 16796 raft_consensus.cc:686] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Becoming Leader. State: Replica: df8dd992665445f2b51ab503c874cea0, State: Running, Role: LEADER
I20260928 05:23:45.405807 16705 raft_consensus.cc:2454] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8dd992665445f2b51ab503c874cea0 in term 1.
I20260928 05:23:45.406056 16796 consensus_queue.cc:237] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } }
I20260928 05:23:45.408679 16264 catalog_manager.cc:5495] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8dd992665445f2b51ab503c874cea0 (127.8.124.193). New cstate: current_term: 1 leader_uuid: "df8dd992665445f2b51ab503c874cea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8dd992665445f2b51ab503c874cea0" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20260928 05:23:45.447495 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.441028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.8.124.193:45039)
0928 05:23:45.441186 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.441193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.441492 (+ 299us) server_negotiation.cc:407] Connection header received
0928 05:23:45.441674 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.441679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.441757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.441846 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:23:45.442238 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.443308 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.445173 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.445816 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.445932 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.446503 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:23:45.446640 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:23:45.446720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.447158 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.447236 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:45.457844 16705 raft_consensus.cc:1261] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:45.457859 16560 raft_consensus.cc:1261] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Refusing update from remote peer df8dd992665445f2b51ab503c874cea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:23:45.458489 16871 consensus_queue.cc:1035] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b60f54ab44fdab111bb78c4ab87a8" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:45.458931 16871 consensus_queue.cc:1035] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19018bf902a46ddb2b74c5f79e510f2" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:23:45.519964 16264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:45.520162 16264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:23:45.522646 16264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6798823038ad4c15b9ef21e20aaf7633: Sending DeleteTablet for 3 replicas of tablet ecedba6ec315460f8aa88683fe1085c5
I20260928 05:23:45.523373 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet ecedba6ec315460f8aa88683fe1085c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60398
I20260928 05:23:45.523502 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet ecedba6ec315460f8aa88683fe1085c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:48432
I20260928 05:23:45.523653 16842 tablet_replica.cc:330] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: stopping tablet replica
I20260928 05:23:45.523756 16843 tablet_replica.cc:330] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: stopping tablet replica
I20260928 05:23:45.523797 16842 raft_consensus.cc:2229] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:23:45.523909 16843 raft_consensus.cc:2229] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:45.524087 16843 raft_consensus.cc:2258] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.524085 16842 raft_consensus.cc:2258] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.525081 16843 ts_tablet_manager.cc:1905] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:45.525311 16842 ts_tablet_manager.cc:1905] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:45.525460 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 16336
I20260928 05:23:45.530761 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet ecedba6ec315460f8aa88683fe1085c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:23:45 UTC) from {username='oryx', principal='oryx/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:39472
I20260928 05:23:45.531071 16844 tablet_replica.cc:330] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: stopping tablet replica
I20260928 05:23:45.531250 16844 raft_consensus.cc:2229] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:23:45.531450 16844 raft_consensus.cc:2258] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:23:45.532634 16844 ts_tablet_manager.cc:1905] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:23:45.533313 16843 ts_tablet_manager.cc:1918] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:45.533521 16843 log.cc:1196] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/ecedba6ec315460f8aa88683fe1085c5
I20260928 05:23:45.533983 16843 ts_tablet_manager.cc:1939] T ecedba6ec315460f8aa88683fe1085c5 P e19018bf902a46ddb2b74c5f79e510f2: Deleting consensus metadata
I20260928 05:23:45.535108 16249 catalog_manager.cc:4841] TS e19018bf902a46ddb2b74c5f79e510f2 (127.8.124.195:41043): tablet ecedba6ec315460f8aa88683fe1085c5 (table test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]) successfully deleted
I20260928 05:23:45.538367 16844 ts_tablet_manager.cc:1918] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:45.538479 16844 log.cc:1196] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/ecedba6ec315460f8aa88683fe1085c5
I20260928 05:23:45.538805 16844 ts_tablet_manager.cc:1939] T ecedba6ec315460f8aa88683fe1085c5 P 214b60f54ab44fdab111bb78c4ab87a8: Deleting consensus metadata
I20260928 05:23:45.539743 16249 catalog_manager.cc:4841] TS 214b60f54ab44fdab111bb78c4ab87a8 (127.8.124.194:44371): tablet ecedba6ec315460f8aa88683fe1085c5 (table test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]) successfully deleted
I20260928 05:23:45.541018 16842 ts_tablet_manager.cc:1918] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:23:45.541136 16842 log.cc:1196] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/ecedba6ec315460f8aa88683fe1085c5
I20260928 05:23:45.541469 16842 ts_tablet_manager.cc:1939] T ecedba6ec315460f8aa88683fe1085c5 P df8dd992665445f2b51ab503c874cea0: Deleting consensus metadata
I20260928 05:23:45.542590 16251 catalog_manager.cc:4841] TS df8dd992665445f2b51ab503c874cea0 (127.8.124.193:45039): tablet ecedba6ec315460f8aa88683fe1085c5 (table test-table [id=e3cb1528abab4a82b7ace555a9dd61a7]) successfully deleted
Sep 28 05:23:45 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.193@KRBTEST.COM
I20260928 05:23:45.552287 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.527231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.8.124.193:45039)
0928 05:23:45.527401 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.527408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.535852 (+ 8444us) server_negotiation.cc:407] Connection header received
0928 05:23:45.536024 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.536030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.536140 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.536259 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:45.537694 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.538799 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.540675 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.541331 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.541424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.546155 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:45.546173 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:45.546194 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:45.546234 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:45.549647 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.550297 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.550306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.550310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.550467 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.550786 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.550790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.550793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.551251 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:45.551565 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.551716 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.551927 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:23:45.552951 16467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:45.672951 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 16336
I20260928 05:23:45.698105 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 16476
Sep 28 05:23:45 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.194@KRBTEST.COM
I20260928 05:23:45.711612 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.698732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.8.124.194:44371)
0928 05:23:45.698908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.698912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.698928 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:45.699084 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.699088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.699160 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.699245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:45.700412 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.701097 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.702310 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.702744 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.702840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.706702 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:45.706719 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:45.706746 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:45.706784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:45.709319 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.709937 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.709944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.709947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.710068 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.710343 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.710346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.710357 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.710690 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:45.710892 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.711153 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.711327 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:23:45.712204 16606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:45.829921 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 16476
I20260928 05:23:45.846829 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 16615
Sep 28 05:23:45 dist-test-slave-515x krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.124.195@KRBTEST.COM
I20260928 05:23:45.860773 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:45.847527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.8.124.195:41043)
0928 05:23:45.847973 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:45.847978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:45.847994 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:23:45.848048 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:45.848052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:45.848107 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:45.848195 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:45.849417 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.850134 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.851353 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:45.851739 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:45.851849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:45.855716 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:45.855730 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:45.855747 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:45.855783 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:45.858248 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.858856 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.858863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.858865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.859021 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:45.859371 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:45.859375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:45.859377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:45.859692 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:45.859990 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:45.860249 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 05:23:45.860476 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
I20260928 05:23:45.861362 16745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:45.978471 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 16615
I20260928 05:23:45.995309 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 16233
I20260928 05:23:45.996421 16305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:46.106594 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 16233
2026-09-28T05:23:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6356 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:46 dist-test-slave-515x krb5kdc[16889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:46 dist-test-slave-515x krb5kdc[16889](info): set up 2 sockets
Sep 28 05:23:46 dist-test-slave-515x krb5kdc[16889](info): commencing operation
krb5kdc: starting...
W20260928 05:23:47.106331 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:47 dist-test-slave-515x krb5kdc[16889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:47.303211 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33407
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:47.611392 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:47.611650 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:47.611747 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:47.618592 16902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:47.618682 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:47.618734 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:47.618788 16902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:47.618832 16902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:47.627982 16902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:47.629089 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:47.630290 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:47.641038 16908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:47.640985 16907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:47.641075 16910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:47.641693 16902 server_base.cc:1034] running on GCE node
I20260928 05:23:47.642411 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:47.643692 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:47.644898 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790573027644833 us; error 49 us; skew 500 ppm
I20260928 05:23:47.645237 16902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T05:23:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1604 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:23:47 dist-test-slave-515x krb5kdc[16916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:23:47 dist-test-slave-515x krb5kdc[16916](info): set up 2 sockets
Sep 28 05:23:47 dist-test-slave-515x krb5kdc[16916](info): commencing operation
krb5kdc: starting...
W20260928 05:23:48.698333 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:23:48 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:23:48.892805 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41039
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:49.197764 16929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:49.198061 16929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:49.198187 16929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:49.205183 16929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:49.205300 16929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:49.205377 16929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:49.205454 16929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:49.205506 16929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:49.215109 16929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:49.216339 16929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:49.217579 16929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:49.228394 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:49.228525 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:49.228327 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:49.229375 16929 server_base.cc:1034] running on GCE node
I20260928 05:23:49.230206 16929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:49.231595 16929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:49.232793 16929 hybrid_clock.cc:648] HybridClock initialized: now 1790573029232744 us; error 38 us; skew 500 ppm
I20260928 05:23:49.233114 16929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:49 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573029, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:49.236023 16929 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:49.237383 16929 webserver.cc:460] Webserver started at http://127.8.124.254:36627/ using document root <none> and password file <none>
I20260928 05:23:49.238076 16929 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:49.238179 16929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:49.238458 16929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:49.241273 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "cff954f4412f4085a80141df962530bd"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "32c3d36f8df88d50ac961c6e2a14186e"
server_key_iv: "d3864c71d74aaceda10a310b53bfa42e"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.242012 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "cff954f4412f4085a80141df962530bd"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "32c3d36f8df88d50ac961c6e2a14186e"
server_key_iv: "d3864c71d74aaceda10a310b53bfa42e"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.246727 16929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:23:49.249926 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:49.251178 16929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:23:49.251325 16929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "cff954f4412f4085a80141df962530bd"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "32c3d36f8df88d50ac961c6e2a14186e"
server_key_iv: "d3864c71d74aaceda10a310b53bfa42e"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.251493 16929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:49.264932 16929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:49.268646 16929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:49.268867 16929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:49.284605 16929 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:41039
I20260928 05:23:49.284626 16995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:41039 every 8 connection(s)
I20260928 05:23:49.285887 16929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:49.289187 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:23:49.289897 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 16929
I20260928 05:23:49.290069 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:23:49.290689 8691 external_mini_cluster.cc:1428] Setting key 18e9f945a7d2a77a86bc3644003e3244
I20260928 05:23:49.301903 16996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Bootstrap starting.
Sep 28 05:23:49 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:49.305413 16996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Neither blocks nor log segments found. Creating new log.
I20260928 05:23:49.306424 16996 log.cc:824] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:49.309275 16996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: No bootstrap required, opened a new log
I20260928 05:23:49.312745 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:49.292555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.8.124.254:41039)
0928 05:23:49.293283 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:49.293293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:49.293336 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:23:49.294874 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:49.294896 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:49.295271 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:49.295577 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:49.297232 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.298567 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.300255 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.300905 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.301036 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:49.305405 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:49.305413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:49.305426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:49.305487 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:49.308665 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:49.309246 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:49.309255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:49.309266 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:49.309459 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:49.309813 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:49.309818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:49.309821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:49.310485 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:49.310750 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:49.311053 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 05:23:49.311461 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":332,"threads_started":1}
I20260928 05:23:49.317472 16996 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:49.317798 16996 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:23:49.317868 16996 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff954f4412f4085a80141df962530bd, State: Initialized, Role: FOLLOWER
I20260928 05:23:49.318303 16996 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:49.318506 16996 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:49.318607 16996 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:49.318720 16996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:23:49.321069 16996 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:49.321497 16996 leader_election.cc:302] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff954f4412f4085a80141df962530bd; no voters:
I20260928 05:23:49.322057 16996 leader_election.cc:288] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:23:49.322145 17001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:23:49.322721 17001 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 LEADER]: Becoming Leader. State: Replica: cff954f4412f4085a80141df962530bd, State: Running, Role: LEADER
I20260928 05:23:49.323196 17001 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:49.323822 16996 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:49.325893 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff954f4412f4085a80141df962530bd. Latest consensus state: current_term: 1 leader_uuid: "cff954f4412f4085a80141df962530bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } } }
I20260928 05:23:49.326318 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:49.326838 17002 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff954f4412f4085a80141df962530bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } } }
I20260928 05:23:49.327055 17002 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:49.328717 17010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:49.332777 17010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:49.340250 17010 catalog_manager.cc:1347] Generated new cluster ID: 8ed64437e0904414ac4e56dc6cb4694f
I20260928 05:23:49.340353 17010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:49.364527 17010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:23:49.365837 17010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:49.372206 17010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Generated new TSK 0
I20260928 05:23:49.372962 17010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:23:49.430716 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:49.737394 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:49.737682 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:49.737814 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:49.744768 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:49.744879 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:49.745026 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:49.754406 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:49.755661 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:49.756873 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:49.769062 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:49.769104 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:49.769164 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:49.770203 17022 server_base.cc:1034] running on GCE node
I20260928 05:23:49.770771 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:49.771651 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:49.772866 17022 hybrid_clock.cc:648] HybridClock initialized: now 1790573029772798 us; error 59 us; skew 500 ppm
I20260928 05:23:49.773155 17022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:49 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573029, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:49.781787 17022 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:23:49.783178 17022 webserver.cc:460] Webserver started at http://127.8.124.193:33595/ using document root <none> and password file <none>
I20260928 05:23:49.783815 17022 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:49.783921 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:49.784196 17022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:49.786975 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "6488c0160af94bb7995cbd7f860eb478"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "c30d2697d10404f9f2e7ab083f7159a1"
server_key_iv: "6232c632137705b4281d5b3f4f06db1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.787689 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "6488c0160af94bb7995cbd7f860eb478"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "c30d2697d10404f9f2e7ab083f7159a1"
server_key_iv: "6232c632137705b4281d5b3f4f06db1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.792502 17022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:23:49.795861 17037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:49.797079 17022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:23:49.797223 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "6488c0160af94bb7995cbd7f860eb478"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "c30d2697d10404f9f2e7ab083f7159a1"
server_key_iv: "6232c632137705b4281d5b3f4f06db1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:49.797389 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:49.814747 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:49.818403 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:49.818651 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:49.819475 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:49.820715 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:49.820801 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:49.820914 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:49.820963 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:49.849769 17022 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:43777
I20260928 05:23:49.849830 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:43777 every 8 connection(s)
I20260928 05:23:49.851009 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:49.866127 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17022
I20260928 05:23:49.866303 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:23:49.867007 8691 external_mini_cluster.cc:1428] Setting key e9270cbdfb2e2ed3d8cd8122155b738b
Sep 28 05:23:49 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573029, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:49.874539 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:49.853295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:40751 (local address 127.8.124.254:41039)
0928 05:23:49.853714 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:49.853719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:49.854736 (+ 1017us) server_negotiation.cc:407] Connection header received
0928 05:23:49.856545 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:49.856551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:49.856635 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:49.856730 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:49.858159 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.858856 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.860711 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.861279 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.861404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:49.868597 (+ 7193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:49.868604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:49.868609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:49.868647 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:49.871629 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:49.872390 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:49.872394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:49.872398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:49.872557 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:49.872947 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:49.872951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:49.872954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:49.873242 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:49.873367 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:49.873997 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 05:23:49.874216 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":200,"threads_started":1}
I20260928 05:23:49.875402 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:49.853830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:40751)
0928 05:23:49.854579 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:49.854622 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:49.856294 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:49.856960 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:49.856968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:49.857446 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:49.858008 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:49.858025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.859008 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.859012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:49.860552 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:49.860566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:49.861703 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:49.861709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:49.861969 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:49.865826 (+ 3857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:49.865853 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:49.868384 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:49.871777 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:49.871785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:49.871802 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:49.872255 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:49.872669 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:49.872673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:49.872676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:49.872846 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:49.873341 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:49.873346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:49.873678 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 05:23:49.873896 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:23:49.874352 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":241,"threads_started":1}
I20260928 05:23:49.876793 17150 heartbeater.cc:347] Connected to a master server at 127.8.124.254:41039
I20260928 05:23:49.877074 17150 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:49.877791 17150 heartbeater.cc:511] Master 127.8.124.254:41039 requested a full tablet report, sending...
I20260928 05:23:49.879789 16961 ts_manager.cc:194] Registered new tserver with Master: 6488c0160af94bb7995cbd7f860eb478 (127.8.124.193:43777)
I20260928 05:23:49.881578 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:40751
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:23:49.912079 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:50.214388 17156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:50.214666 17156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:50.214784 17156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:50.221805 17156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:50.221903 17156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:50.222036 17156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:50.231587 17156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:50.232827 17156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:50.234095 17156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:50.245977 17162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:50.246002 17164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:50.246138 17161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:50.246685 17156 server_base.cc:1034] running on GCE node
I20260928 05:23:50.247220 17156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:50.248066 17156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:50.249266 17156 hybrid_clock.cc:648] HybridClock initialized: now 1790573030249212 us; error 42 us; skew 500 ppm
I20260928 05:23:50.249548 17156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:50 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573030, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:50.252475 17156 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:23:50.253942 17156 webserver.cc:460] Webserver started at http://127.8.124.194:33365/ using document root <none> and password file <none>
I20260928 05:23:50.254575 17156 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:50.254684 17156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:50.254946 17156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:50.257675 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "a69b4ccf63a64b86972c67e2a611fe39"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "89047ff812f1d9732d08d99a89cb6133"
server_key_iv: "6230218b8af2b77157013be49b080322"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.258397 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "a69b4ccf63a64b86972c67e2a611fe39"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "89047ff812f1d9732d08d99a89cb6133"
server_key_iv: "6230218b8af2b77157013be49b080322"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.263108 17156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:23:50.266251 17171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.267460 17156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:50.267596 17156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "a69b4ccf63a64b86972c67e2a611fe39"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "89047ff812f1d9732d08d99a89cb6133"
server_key_iv: "6230218b8af2b77157013be49b080322"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.267771 17156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:50.280143 17156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:50.284211 17156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:50.284451 17156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:50.285355 17156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:50.286650 17156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:50.286722 17156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.286830 17156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:50.286885 17156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.315887 17156 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:33779
I20260928 05:23:50.315963 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:33779 every 8 connection(s)
I20260928 05:23:50.317112 17156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:50.319670 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17156
I20260928 05:23:50.319831 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:23:50.320386 8691 external_mini_cluster.cc:1428] Setting key a32e55d238dbf3590722f3b0a3e14b19
Sep 28 05:23:50 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573030, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:50.336048 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:50.319402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:34737 (local address 127.8.124.254:41039)
0928 05:23:50.319579 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:50.319583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:50.320926 (+ 1343us) server_negotiation.cc:407] Connection header received
0928 05:23:50.322874 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:50.322879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:50.322976 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:50.323056 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:50.324572 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.325303 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.326857 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.327250 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.327354 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:50.330781 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:50.330786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:50.330790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:50.330820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:50.333137 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.333952 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.333956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.333958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.334071 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.334460 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.334463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.334465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.334695 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:50.334795 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:50.335575 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 05:23:50.335736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:23:50.337360 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:50.320022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:34737)
0928 05:23:50.320772 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:50.320809 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:50.322633 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:50.323288 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:50.323297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:50.323813 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:50.324415 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:50.324432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.325449 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.325453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:50.326688 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:50.326702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.327455 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.327462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:50.327671 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:50.328292 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:50.328313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:50.330568 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:50.333266 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:50.333275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:50.333290 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:50.333835 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:50.334160 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:50.334164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:50.334168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:50.334374 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:50.334799 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:50.334807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:50.335204 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 05:23:50.335472 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 05:23:50.335985 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":280,"threads_started":1}
I20260928 05:23:50.338757 17284 heartbeater.cc:347] Connected to a master server at 127.8.124.254:41039
I20260928 05:23:50.339051 17284 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:50.339707 17284 heartbeater.cc:511] Master 127.8.124.254:41039 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:23:50.341042 16961 ts_manager.cc:194] Registered new tserver with Master: a69b4ccf63a64b86972c67e2a611fe39 (127.8.124.194:33779)
I20260928 05:23:50.341882 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:34737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:23:50.364976 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:50.670276 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:50.670616 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:50.670750 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:50.677817 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:50.677927 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:50.678077 17289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:23:50.687609 17289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:50.688803 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:50.690085 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:50.702586 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:50.702634 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:50.702559 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:50.703449 17289 server_base.cc:1034] running on GCE node
I20260928 05:23:50.704147 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:50.705093 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:50.706300 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790573030706243 us; error 54 us; skew 500 ppm
I20260928 05:23:50.706627 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:50 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573030, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:50.709630 17289 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:23:50.711174 17289 webserver.cc:460] Webserver started at http://127.8.124.195:33089/ using document root <none> and password file <none>
I20260928 05:23:50.711860 17289 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:50.711972 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:50.712251 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:23:50.715226 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "bd34f3f876e54645b259f3dd90149a4e"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "81ee550d7c3a59a8b108fe57b356780f"
server_key_iv: "8ed0973e615d85a8b9d52b185d243058"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.715981 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "bd34f3f876e54645b259f3dd90149a4e"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "81ee550d7c3a59a8b108fe57b356780f"
server_key_iv: "8ed0973e615d85a8b9d52b185d243058"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.720906 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:23:50.724217 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.725514 17289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:50.725713 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "bd34f3f876e54645b259f3dd90149a4e"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "81ee550d7c3a59a8b108fe57b356780f"
server_key_iv: "8ed0973e615d85a8b9d52b185d243058"
server_key_version: "encryptionkey@0"
I20260928 05:23:50.725893 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:50.744791 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:50.748669 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:50.748910 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:50.749922 17289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:50.751346 17289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:50.751413 17289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.751498 17289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:50.751539 17289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:50.779232 17289 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:43123
I20260928 05:23:50.779273 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:43123 every 8 connection(s)
I20260928 05:23:50.780354 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:50.782543 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17289
I20260928 05:23:50.782704 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:23:50.783217 8691 external_mini_cluster.cc:1428] Setting key abc47f27561073829b22d47d997c5225
Sep 28 05:23:50 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573030, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:23:50.798223 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:50.782671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:51051 (local address 127.8.124.254:41039)
0928 05:23:50.782813 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:50.782817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:50.784248 (+ 1431us) server_negotiation.cc:407] Connection header received
0928 05:23:50.786355 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:50.786359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:50.786418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:50.786489 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:50.787773 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.788463 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.790017 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.790421 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.790506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:50.793430 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:50.793434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:50.793438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:50.793467 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:50.795574 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.796159 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.796162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.796165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.796280 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.796676 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.796680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.796682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.796939 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:50.797061 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:50.797669 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 05:23:50.797887 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 05:23:50.799042 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:50.783302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:51051)
0928 05:23:50.784098 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:50.784135 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:50.786153 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:50.786664 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:50.786671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:50.787135 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:50.787655 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:50.787671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.788572 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.788575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:50.789884 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:50.789898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.790587 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.790592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:50.790774 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:50.791315 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:50.791338 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:50.793257 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:23:50.795685 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:50.795692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:50.795703 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:50.796080 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:50.796396 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:23:50.796401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:23:50.796405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:23:50.796570 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:23:50.797048 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:23:50.797055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:23:50.797339 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 05:23:50.797558 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 05:23:50.798047 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":299,"threads_started":1}
I20260928 05:23:50.800143 17417 heartbeater.cc:347] Connected to a master server at 127.8.124.254:41039
I20260928 05:23:50.800418 17417 heartbeater.cc:464] Registering TS with master...
I20260928 05:23:50.801101 17417 heartbeater.cc:511] Master 127.8.124.254:41039 requested a full tablet report, sending...
I20260928 05:23:50.802340 16961 ts_manager.cc:194] Registered new tserver with Master: bd34f3f876e54645b259f3dd90149a4e (127.8.124.195:43123)
I20260928 05:23:50.803049 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:51051
I20260928 05:23:50.809795 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
Entry for principal oryx/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab.
WARNING: no policy specified for oryx/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.194@KRBTEST.COM" created.
I20260928 05:23:50.884482 17150 heartbeater.cc:503] Master 127.8.124.254:41039 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
Entry for principal oryx/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab.
WARNING: no policy specified for oryx/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
Entry for principal oryx/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab.
I20260928 05:23:50.928516 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17022
Sep 28 05:23:50 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:23:50.943261 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:50.929581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.8.124.193:43777)
0928 05:23:50.930244 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:50.930251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:50.930277 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:23:50.930336 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:50.930342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:50.930593 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:50.930718 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:50.932050 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.933049 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.934271 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:50.934817 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:50.934909 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:50.938607 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:50.938616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:50.938630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:50.938659 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:50.941011 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.941637 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.941646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.941649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.941757 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:50.942040 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:50.942042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:50.942044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:50.942386 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:50.942601 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:50.942824 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 05:23:50.942994 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":297,"threads_started":1}
I20260928 05:23:50.944402 17145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:51.024358 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17022
I20260928 05:23:51.037133 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17156
Sep 28 05:23:51 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:23:51.051646 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:51.037980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.8.124.194:33779)
0928 05:23:51.038523 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:51.038530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:51.038555 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:23:51.038613 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:51.038619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:51.038824 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:51.038944 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:51.040059 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:51.041042 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:51.042395 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:51.043109 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:51.043207 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:51.046847 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:51.046858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:51.046871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:51.046901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:51.049459 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:51.050029 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:51.050036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:51.050038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:51.050161 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:51.050418 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:51.050421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:51.050423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:51.050720 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:51.050930 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:51.051124 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 05:23:51.051318 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260928 05:23:51.052824 17279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:51.131898 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17156
I20260928 05:23:51.144028 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17289
Sep 28 05:23:51 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:23:51.158805 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:51.144969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.8.124.195:43123)
0928 05:23:51.145538 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:51.145544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:51.145570 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:23:51.145672 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:51.145679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:51.145883 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:51.146006 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:51.147188 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:51.148244 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:51.149440 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:51.150031 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:51.150125 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:51.153954 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:51.153964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:51.153979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:51.154012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:51.156566 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:51.157152 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:51.157158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:51.157160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:51.157274 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:51.157530 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:51.157533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:51.157535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:51.157878 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:51.158096 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:51.158281 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 05:23:51.158463 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":298,"threads_started":1}
I20260928 05:23:51.160095 17412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:51.242388 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17289
I20260928 05:23:51.254813 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 16929
I20260928 05:23:51.255858 16991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:23:51.354493 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 16929
I20260928 05:23:51.370524 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41039
--webserver_interface=127.8.124.254
--webserver_port=36627
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:23:51.671368 17434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:51.671619 17434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:51.671729 17434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:51.678718 17434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:23:51.678807 17434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:51.678870 17434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:51.678943 17434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:23:51.679000 17434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:23:51.688239 17434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:51.689376 17434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:51.690619 17434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:51.701066 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:51.701153 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:51.701133 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:51.701866 17434 server_base.cc:1034] running on GCE node
I20260928 05:23:51.702553 17434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:51.703809 17434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:51.704988 17434 hybrid_clock.cc:648] HybridClock initialized: now 1790573031704940 us; error 39 us; skew 500 ppm
I20260928 05:23:51.705286 17434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:51 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573031, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:51.707994 17434 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:23:51.709300 17434 webserver.cc:460] Webserver started at http://127.8.124.254:36627/ using document root <none> and password file <none>
I20260928 05:23:51.709944 17434 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:51.710059 17434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:51.715031 17434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:23:51.717684 17449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:51.718878 17434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:23:51.719018 17434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "cff954f4412f4085a80141df962530bd"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "32c3d36f8df88d50ac961c6e2a14186e"
server_key_iv: "d3864c71d74aaceda10a310b53bfa42e"
server_key_version: "encryptionkey@0"
I20260928 05:23:51.719869 17434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:51.738996 17434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:51.742681 17434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:51.742902 17434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:51.758978 17434 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:41039
I20260928 05:23:51.758996 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:41039 every 8 connection(s)
I20260928 05:23:51.760296 17434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:23:51.766608 17501 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:23:51.768056 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17434
I20260928 05:23:51.768760 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:43777
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:41039
--webserver_port=33595
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:23:51.769708 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Bootstrap starting.
I20260928 05:23:51.787266 17501 log.cc:824] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:23:51.795703 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:23:51.796161 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Bootstrap complete.
I20260928 05:23:51.804544 17501 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:51.805290 17501 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff954f4412f4085a80141df962530bd, State: Initialized, Role: FOLLOWER
I20260928 05:23:51.805910 17501 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:51.806104 17501 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:23:51.806203 17501 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:23:51.806361 17501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:23:51.809842 17501 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:51.810271 17501 leader_election.cc:302] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff954f4412f4085a80141df962530bd; no voters:
I20260928 05:23:51.810931 17501 leader_election.cc:288] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:23:51.811066 17505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:23:51.812287 17505 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [term 2 LEADER]: Becoming Leader. State: Replica: cff954f4412f4085a80141df962530bd, State: Running, Role: LEADER
I20260928 05:23:51.812850 17505 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } }
I20260928 05:23:51.813144 17501 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:23:51.815346 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff954f4412f4085a80141df962530bd. Latest consensus state: current_term: 2 leader_uuid: "cff954f4412f4085a80141df962530bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } } }
I20260928 05:23:51.815721 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:51.815371 17506 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cff954f4412f4085a80141df962530bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff954f4412f4085a80141df962530bd" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41039 } } }
I20260928 05:23:51.816366 17506 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd [sys.catalog]: This master's current role is: LEADER
I20260928 05:23:51.828378 17511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:23:51.834141 17511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:23:51.835431 17511 catalog_manager.cc:1259] Loaded cluster ID: 8ed64437e0904414ac4e56dc6cb4694f
I20260928 05:23:51.835613 17511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:23:51.839322 17511 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:23:51.840692 17511 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cff954f4412f4085a80141df962530bd: Loaded TSK: 0
I20260928 05:23:51.842494 17511 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:23:52.085071 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:52.085354 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:52.085493 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:52.092717 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:52.092823 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:52.092959 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:23:52.102530 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.193.keytab
--principal=oryx/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:43777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=33595
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:52.103785 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:52.105028 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:52.119028 17530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:52.119009 17527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:52.119251 17528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:52.120204 17503 server_base.cc:1034] running on GCE node
I20260928 05:23:52.120781 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:52.121742 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:52.122961 17503 hybrid_clock.cc:648] HybridClock initialized: now 1790573032122912 us; error 40 us; skew 500 ppm
I20260928 05:23:52.123267 17503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573032, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:52.126395 17503 init.cc:377] Logged in from keytab as oryx/127.8.124.193@KRBTEST.COM (short username oryx)
I20260928 05:23:52.127844 17503 webserver.cc:460] Webserver started at http://127.8.124.193:33595/ using document root <none> and password file <none>
I20260928 05:23:52.128485 17503 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:52.128610 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:52.133874 17503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:23:52.136582 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.138077 17503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:23:52.138230 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "6488c0160af94bb7995cbd7f860eb478"
format_stamp: "Formatted at 2026-09-28 05:23:49 on dist-test-slave-515x"
server_key: "c30d2697d10404f9f2e7ab083f7159a1"
server_key_iv: "6232c632137705b4281d5b3f4f06db1c"
server_key_version: "encryptionkey@0"
I20260928 05:23:52.139123 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:52.160444 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:52.164400 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:52.164651 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:52.165570 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:52.166875 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:52.166965 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.167068 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:52.167137 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.202280 17503 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:43777
I20260928 05:23:52.202358 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:43777 every 8 connection(s)
I20260928 05:23:52.204033 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:23:52.207928 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17503
I20260928 05:23:52.208626 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:33779
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:41039
--webserver_port=33365
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:23:52.597239 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:52.597492 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:52.597616 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:52.604535 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:52.604628 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:52.604780 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:23:52.614090 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.194.keytab
--principal=oryx/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:33779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=33365
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:23:52.615260 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:52.616464 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:23:52.631735 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:52.631831 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:52.632215 17654 server_base.cc:1034] running on GCE node
W20260928 05:23:52.632306 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:52.632894 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:52.633754 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:52.634949 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790573032634879 us; error 57 us; skew 500 ppm
I20260928 05:23:52.635223 17654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573032, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:52.638211 17654 init.cc:377] Logged in from keytab as oryx/127.8.124.194@KRBTEST.COM (short username oryx)
I20260928 05:23:52.639609 17654 webserver.cc:460] Webserver started at http://127.8.124.194:33365/ using document root <none> and password file <none>
I20260928 05:23:52.640206 17654 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:52.640318 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:52.645712 17654 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:23:52.648504 17670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.649963 17654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:52.650116 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "a69b4ccf63a64b86972c67e2a611fe39"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "89047ff812f1d9732d08d99a89cb6133"
server_key_iv: "6230218b8af2b77157013be49b080322"
server_key_version: "encryptionkey@0"
I20260928 05:23:52.651090 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:52.672401 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:52.676262 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:52.676501 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:52.677462 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:52.678735 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:52.678831 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.678929 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:52.678993 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:52.714510 17654 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:33779
I20260928 05:23:52.714550 17782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:33779 every 8 connection(s)
I20260928 05:23:52.715927 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:23:52.720932 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17654
I20260928 05:23:52.721434 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:43123
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:41039
--webserver_port=33089
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:52 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:23:53.210983 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:23:53.211387 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:23:53.211618 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:23:53.224386 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:23:53.224545 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:23:53.224761 17787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
W20260928 05:23:53.233104 17652 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.015s user 0.003s sys 0.008s
I20260928 05:23:53.234988 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:52.207212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:52783)
0928 05:23:52.208086 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:52.208140 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:52.210071 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:52.211588 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:52.211595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:52.212071 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:52.212631 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:52.212649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.214864 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.214869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:52.216085 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:52.216101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.217088 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.217094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:52.217347 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:52.218041 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:52.218082 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.233856 (+1015774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":335,"threads_started":1}
W20260928 05:23:53.234992 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:52.207186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:52783 (local address 127.8.124.254:41039)
0928 05:23:52.208901 (+ 1715us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:52.208915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:52.208961 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:23:52.210658 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:52.210674 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:52.211109 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:52.211465 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:52.212842 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.214702 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.216225 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.216859 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.217006 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.234245 (+1017239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:52783: BlockingRecv error: recv got EOF from 127.8.124.193:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":1262,"threads_started":1}
W20260928 05:23:53.236106 17650 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:41039 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.124.254:41039: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
I20260928 05:23:53.241943 17787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/krb5kdc/oryx_127.8.124.195.keytab
--principal=oryx/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:43123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=33089
--tserver_master_addrs=127.8.124.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:53.243739 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:23:53.244870 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.236734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:56853)
0928 05:23:53.236984 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:53.237000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:53.237111 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:53.237413 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:53.237417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:53.237836 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:53.237998 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.238007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.238940 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.238945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.239871 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.239883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.240639 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.240646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.240799 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.241332 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:53.241349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.244616 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140}
W20260928 05:23:53.245003 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:53.236783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:56853 (local address 127.8.124.254:41039)
0928 05:23:53.236931 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.236937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.237061 (+ 124us) server_negotiation.cc:407] Connection header received
0928 05:23:53.237223 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.237228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.237294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.237370 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.238128 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.238789 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.240007 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.240481 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.240588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.244803 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:56853: BlockingRecv error: recv got EOF from 127.8.124.193:56853 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:23:53.245541 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:53.253167 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.245659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:42409)
0928 05:23:53.245948 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:53.245961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:53.246054 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:53.246340 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:53.246345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:53.246666 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:53.246803 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.246810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.247819 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.247825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.248796 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.248807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.249421 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.249427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.249578 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.250100 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:53.250116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.252930 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208}
W20260928 05:23:53.253310 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:53.245706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:42409 (local address 127.8.124.254:41039)
0928 05:23:53.245853 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.245857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.246023 (+ 166us) server_negotiation.cc:407] Connection header received
0928 05:23:53.246152 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.246157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.246228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.246300 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.246921 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.247697 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.248914 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.249289 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.249387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.253116 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:42409: BlockingRecv error: recv got EOF from 127.8.124.193:42409 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 05:23:53.253688 17650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:23:53.260555 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:53.260505 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:23:53.263281 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:23:53.263473 17787 server_base.cc:1034] running on GCE node
I20260928 05:23:53.264043 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:23:53.264926 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:23:53.266108 17787 hybrid_clock.cc:648] HybridClock initialized: now 1790573033266032 us; error 73 us; skew 500 ppm
I20260928 05:23:53.266381 17787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573033, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:23:53.269523 17787 init.cc:377] Logged in from keytab as oryx/127.8.124.195@KRBTEST.COM (short username oryx)
I20260928 05:23:53.271340 17787 webserver.cc:460] Webserver started at http://127.8.124.195:33089/ using document root <none> and password file <none>
I20260928 05:23:53.272347 17787 fs_manager.cc:359] Metadata directory not provided
I20260928 05:23:53.272504 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:23:53.280102 17787 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:23:53.283640 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:53.285400 17787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:23:53.285585 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "bd34f3f876e54645b259f3dd90149a4e"
format_stamp: "Formatted at 2026-09-28 05:23:50 on dist-test-slave-515x"
server_key: "81ee550d7c3a59a8b108fe57b356780f"
server_key_iv: "8ed0973e615d85a8b9d52b185d243058"
server_key_version: "encryptionkey@0"
I20260928 05:23:53.287055 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:23:53.317432 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:23:53.321563 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:23:53.321826 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:23:53.322853 17787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:23:53.324263 17787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:23:53.324345 17787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:53.324455 17787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:23:53.324518 17787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:23:53.363674 17787 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:43123
I20260928 05:23:53.363724 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:43123 every 8 connection(s)
I20260928 05:23:53.365065 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:23:53.370131 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17787
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:53.402894 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.371174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.8.124.254:41039)
0928 05:23:53.383449 (+ 12275us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.383456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.384202 (+ 746us) server_negotiation.cc:407] Connection header received
0928 05:23:53.384383 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.384389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.384472 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.384595 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.386705 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.387860 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.389883 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.390497 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.390621 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.394291 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:23:53.394305 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:23:53.394335 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:23:53.394392 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:23:53.398864 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:53.399788 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:53.399798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:53.399810 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:53.400061 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:23:53.400463 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:23:53.400468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:23:53.400472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:23:53.401196 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:23:53.401534 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 05:23:53.401979 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 05:23:53.402422 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12124,"thread_start_us":12289,"threads_started":1}
W20260928 05:23:53.837416 17786 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.106s user 0.004s sys 0.013s
W20260928 05:23:53.838668 17785 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:52.718868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:38015 (local address 127.8.124.254:41039)
0928 05:23:52.719351 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:52.719357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:52.720335 (+ 978us) server_negotiation.cc:407] Connection header received
0928 05:23:52.723299 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:52.723304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:52.723377 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:52.723483 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:52.725687 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.726666 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.729160 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.729786 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.729913 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.838336 (+1108423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:38015: BlockingRecv error: recv got EOF from 127.8.124.194:38015 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":230,"threads_started":1}
I20260928 05:23:53.839286 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:52.719286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:38015)
0928 05:23:52.720153 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:52.720209 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:52.722965 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:52.723785 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:52.723799 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:52.724613 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:52.725483 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:52.725510 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.726815 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.726820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:52.729018 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:52.729035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:52.730041 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:52.730049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:52.730393 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:52.731170 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:52.731208 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.838154 (+1106946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
W20260928 05:23:53.840423 17783 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:41039 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.124.254:41039: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:53.850783 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.841071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:57839)
0928 05:23:53.841286 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:53.841308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:53.841435 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:53.841751 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:53.841757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:53.842315 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:53.842554 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.842565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.843685 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.843691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.845381 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.845399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.846113 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.846119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.846303 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.846910 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:53.846934 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.850603 (+ 3669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 05:23:53.851110 17785 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:53.841272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57839 (local address 127.8.124.254:41039)
0928 05:23:53.841413 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.841418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.841433 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:53.841543 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.841546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.841634 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.841714 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.842681 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.843562 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.845518 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.845950 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.846035 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.850844 (+ 4809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:57839: BlockingRecv error: recv got EOF from 127.8.124.194:57839 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:53 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:53.860452 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.851350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:34695)
0928 05:23:53.852039 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:53.852057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:53.852467 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:53.853108 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:53.853114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:53.853512 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:53.853748 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.853763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.854759 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.854763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.855721 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.855732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.856334 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.856340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.856492 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.856974 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:53.856990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:53.860182 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":2446,"threads_started":1}
W20260928 05:23:53.860633 17785 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:53.851706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:34695 (local address 127.8.124.254:41039)
0928 05:23:53.851863 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.851869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.852362 (+ 493us) server_negotiation.cc:407] Connection header received
0928 05:23:53.852607 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.852613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.852797 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.852870 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.853909 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.854616 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.855839 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.856211 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.856306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.860396 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:34695: BlockingRecv error: recv got EOF from 127.8.124.194:34695 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 05:23:53.860918 17783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:54.264614 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:54.254172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:43869)
0928 05:23:54.254873 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:54.254896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:54.255061 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:54.255387 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:54.255392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:54.256000 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:54.256221 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.256234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.257129 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.257135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.258828 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.258844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.259523 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.259530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.259716 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.260327 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:54.260351 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:54.264311 (+ 3960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
W20260928 05:23:54.264714 17785 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:54.254249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:43869 (local address 127.8.124.254:41039)
0928 05:23:54.254439 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:54.254445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:54.254976 (+ 531us) server_negotiation.cc:407] Connection header received
0928 05:23:54.255184 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:54.255188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:54.255267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:54.255339 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:54.256342 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.257009 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.258961 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.259346 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.259433 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.264509 (+ 5076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:43869: BlockingRecv error: recv got EOF from 127.8.124.193:43869 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 05:23:54.370258 17918 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.990s user 0.001s sys 0.012s
W20260928 05:23:54.371222 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:53.368053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:51469 (local address 127.8.124.254:41039)
0928 05:23:53.368288 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:53.368296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:53.369419 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 05:23:53.371540 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:53.371546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:53.371635 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:53.371710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:53.374107 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.375301 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.377848 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.378558 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.378654 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.371037 (+992383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:51469: BlockingRecv error: recv got EOF from 127.8.124.195:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:23:54.371706 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:53.368480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:51469)
0928 05:23:53.369262 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:53.369307 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:53.371199 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:53.372263 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:53.372276 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:53.373005 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:53.373866 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.373893 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.375527 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.375534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.377675 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:53.377691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:53.379007 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:53.379014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:53.379339 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:53.380182 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:53.380219 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:54.370908 (+990689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
W20260928 05:23:54.372573 17916 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:41039 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.124.254:41039: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:54.381083 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:54.373137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:52667)
0928 05:23:54.373292 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:54.373311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:54.373422 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:54.373726 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:54.373730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:54.374067 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:54.374213 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.374223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.375228 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.375234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.376176 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.376187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.376888 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.376895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.377036 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.377631 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:54.377656 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:54.380825 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:23:54.381245 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:54.373248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:52667 (local address 127.8.124.254:41039)
0928 05:23:54.373428 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:54.373434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:54.373451 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:23:54.373524 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:54.373529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:54.373591 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:54.373701 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:54.374345 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.375096 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.376303 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.376734 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.376838 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.381019 (+ 4181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:52667: BlockingRecv error: recv got EOF from 127.8.124.195:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:54.388276 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:54.381706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:40971)
0928 05:23:54.381848 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:54.381865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:54.381936 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:54.382158 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:54.382162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:54.382413 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:54.382543 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.382550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.383414 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.383417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.384307 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.384316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.384884 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.384888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.385005 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.385428 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:54.385442 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:54.388110 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:23:54.388455 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:54.381776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:40971 (local address 127.8.124.254:41039)
0928 05:23:54.381909 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:54.381912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:54.381927 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:23:54.382014 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:54.382017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:54.382067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:54.382136 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:54.382647 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.383318 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.384401 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.384786 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.384866 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.388267 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:40971: BlockingRecv error: recv got EOF from 127.8.124.195:40971 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:23:54.388499 17916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:54 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:54.873744 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:54.861412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:50813)
0928 05:23:54.861920 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:54.861936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:54.862051 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:54.862387 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:54.862390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:54.862767 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:54.862909 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.862919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.863845 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.863848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.864795 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:54.864806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.865434 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.865438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:54.865560 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.870284 (+ 4724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:54.870301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:54.873380 (+ 3079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":311,"threads_started":1}
W20260928 05:23:54.873833 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:54.861513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:50813 (local address 127.8.124.254:41039)
0928 05:23:54.861703 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:54.861708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:54.862006 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:23:54.862182 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:54.862186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:54.862263 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:54.862345 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:54.863024 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.863728 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.864931 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:54.865304 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:54.865396 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:54.873595 (+ 8199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:50813: BlockingRecv error: recv got EOF from 127.8.124.194:50813 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:55.273366 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:55.265448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:49921)
0928 05:23:55.265910 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:55.265926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:55.266038 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:55.266295 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:55.266298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:55.266708 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:55.266878 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.266886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.267795 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.267799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.268708 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.268719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.269299 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.269303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.269408 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.269972 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:55.269988 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:55.273046 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":268,"threads_started":1}
W20260928 05:23:55.273443 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:55.265498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:49921 (local address 127.8.124.254:41039)
0928 05:23:55.265708 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:55.265715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:55.265992 (+ 277us) server_negotiation.cc:407] Connection header received
0928 05:23:55.266135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:55.266138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:55.266194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:55.266269 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:55.266987 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.267683 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.268803 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.269199 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.269279 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.273222 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:49921: BlockingRecv error: recv got EOF from 127.8.124.193:49921 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:55.396921 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:55.389026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:47091)
0928 05:23:55.389541 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:55.389565 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:55.389711 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:55.389965 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:55.389968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:55.390331 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:55.390484 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.390492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.391381 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.391384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.392393 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.392405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.393055 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.393061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.393197 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.393714 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:55.393730 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:55.396694 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":263,"threads_started":1}
W20260928 05:23:55.397061 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:55.389067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47091 (local address 127.8.124.254:41039)
0928 05:23:55.389229 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:55.389235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:55.389662 (+ 427us) server_negotiation.cc:407] Connection header received
0928 05:23:55.389793 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:55.389797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:55.389865 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:55.389941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:55.390582 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.391293 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.392534 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.392902 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.393006 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.396857 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:47091: BlockingRecv error: recv got EOF from 127.8.124.195:47091 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:55 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:55.882810 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:55.874660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:56667)
0928 05:23:55.875266 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:55.875285 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:55.875412 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:55.875737 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:55.875741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:55.876206 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:55.876377 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.876387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.877316 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.877319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.878238 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:55.878251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.878854 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.878858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:55.878986 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.879542 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:55.879560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:55.882564 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":376,"threads_started":1}
W20260928 05:23:55.882906 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:55.874731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:56667 (local address 127.8.124.254:41039)
0928 05:23:55.874894 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:55.874899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:55.875366 (+ 467us) server_negotiation.cc:407] Connection header received
0928 05:23:55.875532 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:55.875535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:55.875610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:55.875689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:55.876496 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.877190 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.878353 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:55.878746 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:55.878829 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:55.882734 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:56667: BlockingRecv error: recv got EOF from 127.8.124.194:56667 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:56.282214 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:56.274393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:59287)
0928 05:23:56.274878 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:56.274897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:56.275004 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:56.275324 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:56.275329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:56.275682 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:56.275851 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.275859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.276771 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.276774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.277687 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.277700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.278320 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.278324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.278438 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.278987 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:56.279003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:56.281987 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
W20260928 05:23:56.282393 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:56.274513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:59287 (local address 127.8.124.254:41039)
0928 05:23:56.274710 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:56.274723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:56.274961 (+ 238us) server_negotiation.cc:407] Connection header received
0928 05:23:56.275097 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:56.275102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:56.275178 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:56.275275 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:56.275958 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.276672 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.277794 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.278221 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.278308 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.282181 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:59287: BlockingRecv error: recv got EOF from 127.8.124.193:59287 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:56.406790 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:56.397864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:44109)
0928 05:23:56.398384 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:56.398408 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:56.398524 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:56.398794 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:56.398797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:56.399166 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:56.399347 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.399355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.400327 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.400331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.401259 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.401270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.402154 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.402161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.402326 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.402923 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:56.402945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:56.406545 (+ 3600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
W20260928 05:23:56.406908 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:56.397942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:44109 (local address 127.8.124.254:41039)
0928 05:23:56.398107 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:56.398111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:56.398493 (+ 382us) server_negotiation.cc:407] Connection header received
0928 05:23:56.398611 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:56.398615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:56.398688 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:56.398768 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:56.399479 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.400198 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.401422 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.402001 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.402114 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.406711 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:44109: BlockingRecv error: recv got EOF from 127.8.124.195:44109 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:56 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:56.891939 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:56.883560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:45557)
0928 05:23:56.884054 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:56.884069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:56.884194 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:56.884487 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:56.884492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:56.884846 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:56.885007 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.885017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.886096 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.886100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.886984 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:56.886995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.887641 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.887646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:56.887808 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.888351 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:56.888374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:56.891641 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
W20260928 05:23:56.892060 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:56.883657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:45557 (local address 127.8.124.254:41039)
0928 05:23:56.883848 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:56.883854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:56.884147 (+ 293us) server_negotiation.cc:407] Connection header received
0928 05:23:56.884300 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:56.884305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:56.884370 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:56.884445 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:56.885151 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.885968 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.887109 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:56.887520 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:56.887617 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:56.891841 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:45557: BlockingRecv error: recv got EOF from 127.8.124.194:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:57.291100 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:57.283090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:51763)
0928 05:23:57.283515 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:57.283530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:57.283644 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:57.283900 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:57.283903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:57.284240 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:57.284389 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.284397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.285302 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.285305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.286304 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.286316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.286993 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.286997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.287107 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.287655 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:57.287671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:57.290786 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
W20260928 05:23:57.291216 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:57.283174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:51763 (local address 127.8.124.254:41039)
0928 05:23:57.283380 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:57.283385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:57.283596 (+ 211us) server_negotiation.cc:407] Connection header received
0928 05:23:57.283741 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:57.283744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:57.283804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:57.283871 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:57.284504 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.285209 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.286470 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.286884 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.286977 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.290995 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:51763: BlockingRecv error: recv got EOF from 127.8.124.193:51763 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:57.416290 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:57.407671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:53157)
0928 05:23:57.408175 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:57.408194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:57.408341 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:57.408727 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:57.408732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:57.409174 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:57.409366 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.409379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.410443 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.410447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.411341 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.411353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.412059 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.412063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.412199 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.412771 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:57.412787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:57.416060 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":248,"threads_started":1}
W20260928 05:23:57.416486 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:57.407762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:53157 (local address 127.8.124.254:41039)
0928 05:23:57.407950 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:57.407956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:57.408314 (+ 358us) server_negotiation.cc:407] Connection header received
0928 05:23:57.408461 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:57.408466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:57.408572 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:57.408678 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:57.409519 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.410284 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.411515 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.411934 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.412025 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.416259 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:53157: BlockingRecv error: recv got EOF from 127.8.124.195:53157 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:57 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:57.901126 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:57.892904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:57863)
0928 05:23:57.893390 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:57.893406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:57.893520 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:57.893947 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:57.893952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:57.894306 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:57.894472 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.894482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.895449 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.895454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.896433 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:57.896445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.897087 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.897093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:57.897246 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.897833 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:57.897851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:57.900885 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
W20260928 05:23:57.901382 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:57.893000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57863 (local address 127.8.124.254:41039)
0928 05:23:57.893195 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:57.893200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:57.893491 (+ 291us) server_negotiation.cc:407] Connection header received
0928 05:23:57.893648 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:57.893653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:57.893733 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:57.893847 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:57.894613 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.895317 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.896576 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:57.896960 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:57.897061 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:57.901107 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:57863: BlockingRecv error: recv got EOF from 127.8.124.194:57863 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:58.299825 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:58.291874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:48389)
0928 05:23:58.292331 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:58.292346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:58.292451 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:58.292779 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:58.292782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:58.293170 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:58.293314 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.293322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.294309 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.294313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.295199 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.295209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.295907 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.295911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.296024 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.296577 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:58.296594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:58.299587 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
W20260928 05:23:58.299948 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:58.291958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:48389 (local address 127.8.124.254:41039)
0928 05:23:58.292126 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:58.292131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:58.292412 (+ 281us) server_negotiation.cc:407] Connection header received
0928 05:23:58.292573 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:58.292577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:58.292662 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:58.292761 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:58.293435 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.294200 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.295347 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.295803 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.295903 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.299763 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:48389: BlockingRecv error: recv got EOF from 127.8.124.193:48389 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:58.425350 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:58.417243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:47189)
0928 05:23:58.417756 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:58.417772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:58.417888 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:58.418197 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:58.418202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:58.418576 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:58.418737 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.418747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.419665 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.419671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.420624 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.420636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.421265 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.421270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.421425 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.421967 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:58.421984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:58.425098 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":246,"threads_started":1}
W20260928 05:23:58.425449 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:58.417301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47189 (local address 127.8.124.254:41039)
0928 05:23:58.417661 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:58.417666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:58.417838 (+ 172us) server_negotiation.cc:407] Connection header received
0928 05:23:58.417989 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:58.417994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:58.418071 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:58.418155 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:58.418862 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.419540 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.420745 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.421142 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.421240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.425275 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:47189: BlockingRecv error: recv got EOF from 127.8.124.195:47189 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:58 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:58.909931 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:58.901919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:59885)
0928 05:23:58.902358 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:58.902374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:58.902494 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:58.902777 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:58.902781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:58.903189 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:58.903335 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.903344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.904305 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.904311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.905210 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:58.905222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.905924 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.905930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:58.906115 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.906616 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:58.906632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:58.909630 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":208,"threads_started":1}
W20260928 05:23:58.910053 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:58.902110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:59885 (local address 127.8.124.254:41039)
0928 05:23:58.902302 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:58.902309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:58.902441 (+ 132us) server_negotiation.cc:407] Connection header received
0928 05:23:58.902600 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:58.902605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:58.902665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:58.902739 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:58.903507 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.904183 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.905343 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:58.905798 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:58.905899 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:58.909833 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:59885: BlockingRecv error: recv got EOF from 127.8.124.194:59885 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:59.309521 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:59.300724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:33481)
0928 05:23:59.301137 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:59.301153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:59.301264 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:59.301544 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:59.301547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:59.302050 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:59.302210 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.302220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.303469 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.303473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.304394 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.304406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.305064 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.305069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.305187 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.305773 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:59.305790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:59.309205 (+ 3415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":213,"threads_started":1}
W20260928 05:23:59.309661 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:59.300812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:33481 (local address 127.8.124.254:41039)
0928 05:23:59.301026 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:59.301031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:59.301220 (+ 189us) server_negotiation.cc:407] Connection header received
0928 05:23:59.301357 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:59.301360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:59.301423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:59.301525 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:59.302348 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.303318 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.304518 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.304933 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.305026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.309391 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:33481: BlockingRecv error: recv got EOF from 127.8.124.193:33481 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:59.434666 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:59.426265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:46141)
0928 05:23:59.426755 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:59.426771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:59.426880 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:59.427162 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:59.427165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:59.427576 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:59.427719 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.427727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.428635 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.428639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.429674 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.429686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.430409 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.430414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.430585 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.431088 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:59.431105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:59.434338 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":289,"threads_started":1}
W20260928 05:23:59.434778 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:59.426329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:46141 (local address 127.8.124.254:41039)
0928 05:23:59.426514 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:59.426521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:59.426838 (+ 317us) server_negotiation.cc:407] Connection header received
0928 05:23:59.426975 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:59.426979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:59.427048 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:59.427136 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:59.427819 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.428515 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.429797 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.430261 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.430352 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.434541 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:46141: BlockingRecv error: recv got EOF from 127.8.124.195:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:23:59 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:23:59.922480 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:23:59.910921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:47135)
0928 05:23:59.911401 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 05:23:59.911419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:23:59.911576 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:23:59.911862 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:23:59.911866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:23:59.912258 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:23:59.912416 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.912425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.913425 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.913430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.914404 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:23:59.914416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.915047 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.915053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:23:59.915215 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.919254 (+ 4039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:23:59.919271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:23:59.922262 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":217,"threads_started":1}
W20260928 05:23:59.922744 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:23:59.910944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:47135 (local address 127.8.124.254:41039)
0928 05:23:59.911123 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:23:59.911129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:23:59.911489 (+ 360us) server_negotiation.cc:407] Connection header received
0928 05:23:59.911688 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:23:59.911692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:23:59.911748 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:23:59.911821 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:23:59.912543 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.913274 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.914527 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:23:59.914913 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:23:59.915016 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:23:59.922524 (+ 7508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:47135: BlockingRecv error: recv got EOF from 127.8.124.194:47135 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:00.318602 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:00.310449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:55745)
0928 05:24:00.311000 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:00.311015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:00.311139 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:00.311438 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:00.311443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:00.311858 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:00.312017 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.312026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.312998 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.313004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.313999 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.314011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.314662 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.314667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.314813 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.315309 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:00.315325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:00.318268 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":260,"threads_started":1}
W20260928 05:24:00.318686 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:00.310563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:55745 (local address 127.8.124.254:41039)
0928 05:24:00.310704 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:00.310710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:00.311108 (+ 398us) server_negotiation.cc:407] Connection header received
0928 05:24:00.311257 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:00.311262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:00.311319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:00.311393 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:00.312165 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.312875 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.314128 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.314528 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.314627 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.318465 (+ 3838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:55745: BlockingRecv error: recv got EOF from 127.8.124.193:55745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:00.443701 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:00.435606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:58537)
0928 05:24:00.436102 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:00.436120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:00.436243 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:00.436532 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:00.436536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:00.436914 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:00.437074 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.437083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.438106 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.438111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.439018 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.439029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.439674 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.439680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.439837 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.440334 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:00.440349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:00.443476 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
W20260928 05:24:00.443849 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:00.435699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:58537 (local address 127.8.124.254:41039)
0928 05:24:00.435880 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:00.435887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:00.436195 (+ 308us) server_negotiation.cc:407] Connection header received
0928 05:24:00.436353 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:00.436358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:00.436420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:00.436494 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:00.437203 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.437974 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.439147 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.439535 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.439634 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.443655 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:58537: BlockingRecv error: recv got EOF from 127.8.124.195:58537 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:00 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:00.931259 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:00.923384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:46297)
0928 05:24:00.923868 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:00.923883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:00.923988 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:00.924241 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:00.924244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:00.924668 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:00.924831 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.924841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.925757 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.925761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.926668 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:00.926678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.927284 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.927287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:00.927393 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.927923 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:00.927939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:00.931014 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":213,"threads_started":1}
W20260928 05:24:00.931372 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:00.923444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:46297 (local address 127.8.124.254:41039)
0928 05:24:00.923633 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:00.923641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:00.923948 (+ 307us) server_negotiation.cc:407] Connection header received
0928 05:24:00.924072 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:00.924075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:00.924150 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:00.924221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:00.924929 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.925664 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.926771 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:00.927182 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:00.927268 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:00.931198 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:46297: BlockingRecv error: recv got EOF from 127.8.124.194:46297 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:01.328047 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:01.319588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:55869)
0928 05:24:01.320148 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:01.320162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:01.320286 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:01.320597 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:01.320602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:01.320986 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:01.321149 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.321158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.322168 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.322173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.323140 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.323152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.323872 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.323879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.324034 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.324599 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:01.324617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:01.327799 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":321,"threads_started":1}
W20260928 05:24:01.328254 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:01.319634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:55869 (local address 127.8.124.254:41039)
0928 05:24:01.319794 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:01.319801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:01.320242 (+ 441us) server_negotiation.cc:407] Connection header received
0928 05:24:01.320400 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:01.320405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:01.320464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:01.320538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:01.321283 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.322033 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.323257 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.323711 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.323813 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.327987 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:55869: BlockingRecv error: recv got EOF from 127.8.124.193:55869 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:01.452936 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:01.444574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:57203)
0928 05:24:01.445119 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:01.445134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:01.445254 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:01.445531 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:01.445535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:01.446012 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:01.446176 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.446185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.447129 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.447135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.448052 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.448062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.448709 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.448715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.448866 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.449389 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:01.449405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:01.452696 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":256,"threads_started":1}
W20260928 05:24:01.453030 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:01.444740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:57203 (local address 127.8.124.254:41039)
0928 05:24:01.444926 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:01.444932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:01.445211 (+ 279us) server_negotiation.cc:407] Connection header received
0928 05:24:01.445357 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:01.445362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:01.445418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:01.445493 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:01.446306 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.447002 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.448174 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.448579 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.448704 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.452846 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:57203: BlockingRecv error: recv got EOF from 127.8.124.195:57203 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:01 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:01.940599 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:01.932108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:52023)
0928 05:24:01.932650 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:01.932665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:01.932790 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:01.933079 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:01.933084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:01.933451 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:01.933633 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.933646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.934733 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.934738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.935644 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:01.935656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.936324 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.936330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:01.936485 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.937003 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:01.937019 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:01.940234 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260928 05:24:01.940618 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:01.932214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:52023 (local address 127.8.124.254:41039)
0928 05:24:01.932387 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:01.932393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:01.932743 (+ 350us) server_negotiation.cc:407] Connection header received
0928 05:24:01.932900 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:01.932904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:01.932961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:01.933037 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:01.933766 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.934599 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.935775 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:01.936165 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:01.936265 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:01.940404 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:52023: BlockingRecv error: recv got EOF from 127.8.124.194:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:02.339686 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:02.329079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:39857)
0928 05:24:02.329585 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:02.329628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:02.329759 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:02.330081 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:02.330086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:02.330596 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:02.330794 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.330806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.331911 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.331917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.333294 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.333310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.334331 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.334338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.334514 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.335128 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:02.335153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:02.339424 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":236,"threads_started":1}
W20260928 05:24:02.339902 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:02.329199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:39857 (local address 127.8.124.254:41039)
0928 05:24:02.329521 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:02.329527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:02.329689 (+ 162us) server_negotiation.cc:407] Connection header received
0928 05:24:02.329890 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:02.329893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:02.329951 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:02.330024 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:02.330906 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.331781 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.333437 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.334187 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.334303 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.339660 (+ 5357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:39857: BlockingRecv error: recv got EOF from 127.8.124.193:39857 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:02.461819 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:02.453895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:52675)
0928 05:24:02.454341 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:02.454356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:02.454470 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:02.454718 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:02.454721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:02.455129 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:02.455283 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.455293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.456249 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.456253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.457174 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.457186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.457894 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.457898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.458017 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.458596 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:02.458613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:02.461530 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
W20260928 05:24:02.461920 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:02.453926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:52675 (local address 127.8.124.254:41039)
0928 05:24:02.454106 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:02.454113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:02.454423 (+ 310us) server_negotiation.cc:407] Connection header received
0928 05:24:02.454555 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:02.454559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:02.454617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:02.454693 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:02.455408 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.456126 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.457292 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.457769 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.457857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.461725 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:52675: BlockingRecv error: recv got EOF from 127.8.124.195:52675 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:02 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:02.949879 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:02.941553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:37253)
0928 05:24:02.942051 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:02.942066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:02.942178 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:02.942497 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:02.942502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:02.942885 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:02.943042 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.943052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.944001 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.944006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.944938 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:02.944950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.945557 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.945562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:02.945703 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.946259 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:02.946278 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:02.949655 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":248,"threads_started":1}
W20260928 05:24:02.949980 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:02.941695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:37253 (local address 127.8.124.254:41039)
0928 05:24:02.941856 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:02.941861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:02.942137 (+ 276us) server_negotiation.cc:407] Connection header received
0928 05:24:02.942279 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:02.942282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:02.942344 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:02.942431 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:02.943153 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.943877 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.945042 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:02.945427 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:02.945511 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:02.949792 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:37253: BlockingRecv error: recv got EOF from 127.8.124.194:37253 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:03.349138 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:03.340785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:52251)
0928 05:24:03.341338 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:03.341356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:03.341493 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:03.341848 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:03.341853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:03.342320 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:03.342484 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.342493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.343533 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.343538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.344568 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.344587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.345216 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.345221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.345378 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.345917 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:03.345933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:03.348902 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":286,"threads_started":1}
W20260928 05:24:03.349263 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:03.340896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:52251 (local address 127.8.124.254:41039)
0928 05:24:03.341098 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:03.341105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:03.341436 (+ 331us) server_negotiation.cc:407] Connection header received
0928 05:24:03.341635 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:03.341641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:03.341713 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:03.341805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:03.342637 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.343414 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.344695 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.345090 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.345187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.349061 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:52251: BlockingRecv error: recv got EOF from 127.8.124.193:52251 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:03.470733 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:03.462657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:36463)
0928 05:24:03.463121 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:03.463137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:03.463242 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:03.463486 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:03.463489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:03.463847 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:03.464007 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.464016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.464947 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.464952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.465920 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.465931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.466536 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.466541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.466655 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.467163 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:03.467179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:03.470443 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
W20260928 05:24:03.470831 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:03.462734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:36463 (local address 127.8.124.254:41039)
0928 05:24:03.462895 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:03.462900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:03.463198 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:24:03.463317 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:03.463320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:03.463380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:03.463453 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:03.464108 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.464810 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.466030 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.466423 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.466510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.470606 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:36463: BlockingRecv error: recv got EOF from 127.8.124.195:36463 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:03 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:03.959218 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:03.950834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:33589)
0928 05:24:03.951301 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:03.951317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:03.951471 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:03.951730 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:03.951734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:03.952078 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:03.952228 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.952237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.953298 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.953303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.954406 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:03.954420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.955195 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.955199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:03.955317 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.955848 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:03.955865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:03.958906 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
W20260928 05:24:03.959308 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:03.950831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:33589 (local address 127.8.124.254:41039)
0928 05:24:03.951005 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:03.951009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:03.951469 (+ 460us) server_negotiation.cc:407] Connection header received
0928 05:24:03.951561 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:03.951565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:03.951626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:03.951709 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:03.952357 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.953140 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.954544 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:03.955023 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:03.955138 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:03.959090 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:33589: BlockingRecv error: recv got EOF from 127.8.124.194:33589 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:04.358249 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:04.349924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:50039)
0928 05:24:04.350375 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:04.350390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:04.350513 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:04.350819 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:04.350824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:04.351208 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:04.351375 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.351384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.352344 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.352350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.353264 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.353275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.354014 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.354019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.354174 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.354699 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:04.354736 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:04.357954 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
W20260928 05:24:04.358357 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:04.350074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:50039 (local address 127.8.124.254:41039)
0928 05:24:04.350258 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:04.350264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:04.350453 (+ 189us) server_negotiation.cc:407] Connection header received
0928 05:24:04.350625 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:04.350630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:04.350688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:04.350761 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:04.351497 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.352230 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.353383 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.353878 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.353974 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.358127 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:50039: BlockingRecv error: recv got EOF from 127.8.124.193:50039 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:04.479792 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:04.471640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:53969)
0928 05:24:04.472135 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:04.472155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:04.472277 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:04.472570 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:04.472575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:04.472966 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:04.473130 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.473140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.474135 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.474141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.475159 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.475171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.475804 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.475810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.475960 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.476530 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:04.476546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:04.479487 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":260,"threads_started":1}
W20260928 05:24:04.479864 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:04.471693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:53969 (local address 127.8.124.254:41039)
0928 05:24:04.471861 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:04.471867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:04.472221 (+ 354us) server_negotiation.cc:407] Connection header received
0928 05:24:04.472391 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:04.472396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:04.472454 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:04.472527 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:04.473263 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.474007 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.475289 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.475672 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.475769 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.479666 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:53969: BlockingRecv error: recv got EOF from 127.8.124.195:53969 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:04 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:04.973253 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:04.960346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:51471)
0928 05:24:04.960821 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:04.960837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:04.960965 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:04.961248 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:04.961253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:04.961699 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:04.961879 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.961889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.962870 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.962876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.963781 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:04.963793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.964486 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.964492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:04.964644 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.969509 (+ 4865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:04.969528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:04.972949 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":269,"threads_started":1}
W20260928 05:24:04.973511 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:04.960444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:51471 (local address 127.8.124.254:41039)
0928 05:24:04.960623 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:04.960628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:04.960914 (+ 286us) server_negotiation.cc:407] Connection header received
0928 05:24:04.961071 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:04.961076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:04.961134 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:04.961216 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:04.962029 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.962748 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.963913 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:04.964349 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:04.964457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:04.973266 (+ 8809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:51471: BlockingRecv error: recv got EOF from 127.8.124.194:51471 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:05.368286 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:05.359226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:38417)
0928 05:24:05.360053 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:05.360073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:05.360212 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:05.360538 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:05.360541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:05.360908 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:05.361068 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.361077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.362139 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.362143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.363036 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.363047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.363721 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.363725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.363845 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.364450 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:05.364471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:05.367970 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":567,"threads_started":1}
W20260928 05:24:05.368393 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:05.359303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:38417 (local address 127.8.124.254:41039)
0928 05:24:05.359568 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:05.359575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:05.360226 (+ 651us) server_negotiation.cc:407] Connection header received
0928 05:24:05.360325 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:05.360331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:05.360392 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:05.360500 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:05.361188 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.361996 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.363169 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.363590 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.363678 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.368173 (+ 4495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:38417: BlockingRecv error: recv got EOF from 127.8.124.193:38417 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:05.488981 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:05.480623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:53927)
0928 05:24:05.481053 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:05.481075 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:05.481193 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:05.481474 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:05.481478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:05.481881 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:05.482060 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.482078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.483036 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.483042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.483933 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.483944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.484593 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.484599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.484756 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.485336 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:05.485354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:05.488740 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
W20260928 05:24:05.489131 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:05.480697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:53927 (local address 127.8.124.254:41039)
0928 05:24:05.480863 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:05.480869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:05.481147 (+ 278us) server_negotiation.cc:407] Connection header received
0928 05:24:05.481295 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:05.481299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:05.481355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:05.481430 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:05.482199 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.482917 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.484059 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.484461 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.484571 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.488924 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:53927: BlockingRecv error: recv got EOF from 127.8.124.195:53927 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:05 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:05.981942 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:05.974124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:59667)
0928 05:24:05.974762 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:05.974777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:05.974880 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:05.975195 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:05.975199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:05.975552 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:05.975726 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.975735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.976640 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.976643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.977526 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:05.977537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.978228 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.978233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:05.978354 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.978884 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:05.978901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:05.981715 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":404,"threads_started":1}
W20260928 05:24:05.982103 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:05.974184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:59667 (local address 127.8.124.254:41039)
0928 05:24:05.974382 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:05.974389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:05.974852 (+ 463us) server_negotiation.cc:407] Connection header received
0928 05:24:05.974987 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:05.974992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:05.975071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:05.975149 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:05.975837 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.976533 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.977685 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:05.978098 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:05.978185 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:05.981909 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:59667: BlockingRecv error: recv got EOF from 127.8.124.194:59667 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:06.377570 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:06.369260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:50647)
0928 05:24:06.369708 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:06.369723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:06.369846 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:06.370174 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:06.370179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:06.370649 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:06.370787 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.370796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.371789 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.371795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.372697 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.372708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.373377 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.373383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.373564 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.374102 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:06.374119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:06.377239 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
W20260928 05:24:06.377684 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:06.369305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:50647 (local address 127.8.124.254:41039)
0928 05:24:06.369468 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:06.369474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:06.369795 (+ 321us) server_negotiation.cc:407] Connection header received
0928 05:24:06.369963 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:06.369990 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:06.370053 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:06.370128 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:06.370921 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.371666 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.372826 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.373250 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.373353 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.377434 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:50647: BlockingRecv error: recv got EOF from 127.8.124.193:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:06.498137 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:06.489799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:51013)
0928 05:24:06.490293 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:06.490308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:06.490428 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:06.490684 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:06.490687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:06.491036 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:06.491224 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.491233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.492149 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.492152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.493142 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.493154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.493885 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.493889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.494015 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.494590 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:06.494606 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:06.497795 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
W20260928 05:24:06.498190 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:06.489850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:51013 (local address 127.8.124.254:41039)
0928 05:24:06.489994 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:06.489999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:06.490375 (+ 376us) server_negotiation.cc:407] Connection header received
0928 05:24:06.490509 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:06.490512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:06.490584 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:06.490659 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:06.491324 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.492054 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.493251 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.493747 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.493838 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.497986 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:51013: BlockingRecv error: recv got EOF from 127.8.124.195:51013 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:06 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:06.991289 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:06.982715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:36407)
0928 05:24:06.983131 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:06.983150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:06.983273 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:06.983583 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:06.983588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:06.984053 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:06.984232 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.984246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.985526 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.985532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.986485 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:06.986498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.987200 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.987205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:06.987340 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.987903 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:06.987920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:06.991077 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
W20260928 05:24:06.991456 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:06.982884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:36407 (local address 127.8.124.254:41039)
0928 05:24:06.983115 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:06.983121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:06.983220 (+ 99us) server_negotiation.cc:407] Connection header received
0928 05:24:06.983381 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:06.983385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:06.983441 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:06.983516 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:06.984379 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.985384 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.986628 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:06.987059 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:06.987150 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:06.991243 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:36407: BlockingRecv error: recv got EOF from 127.8.124.194:36407 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.193@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:07.387239 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:07.378796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.193:54397)
0928 05:24:07.379278 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:07.379293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:07.379414 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:07.379717 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:07.379722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:07.380081 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:07.380260 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.380269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.381291 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.381297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.382253 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.382265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.382931 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.382937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.383103 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:07.383646 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:07.383663 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:07.386934 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":266,"threads_started":1}
W20260928 05:24:07.387326 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:07.378878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:54397 (local address 127.8.124.254:41039)
0928 05:24:07.379069 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:07.379075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:07.379369 (+ 294us) server_negotiation.cc:407] Connection header received
0928 05:24:07.379523 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:07.379528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:07.379596 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:07.379672 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:07.380404 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.381100 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.382378 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.382801 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.382913 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:07.387136 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.193:54397: BlockingRecv error: recv got EOF from 127.8.124.193:54397 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.195@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:07.507297 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:07.499041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.195:59091)
0928 05:24:07.499452 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:07.499470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:07.499601 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:07.499883 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:07.499888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:07.500284 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:07.500416 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.500425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.501379 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.501385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.502641 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.502655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.503381 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.503385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.503497 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:07.504013 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:07.504028 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:07.507088 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":201,"threads_started":1}
W20260928 05:24:07.507471 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:07.499093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:59091 (local address 127.8.124.254:41039)
0928 05:24:07.499395 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:07.499401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:07.499537 (+ 136us) server_negotiation.cc:407] Connection header received
0928 05:24:07.499707 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:07.499712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:07.499769 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:07.499845 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:07.500546 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.501251 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.502789 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.503263 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.503376 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:07.507259 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.195:59091: BlockingRecv error: recv got EOF from 127.8.124.195:59091 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:24:07 dist-test-slave-515x krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.124.194@KRBTEST.COM for oryx/127.8.124.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:24:08.000304 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:07.992284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:41039 (local address 127.8.124.194:32827)
0928 05:24:07.992734 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:07.992749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:07.992852 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:07.993147 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:07.993150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:07.993525 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:07.993732 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.993742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.994673 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.994677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.995562 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:07.995573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.996309 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.996314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:07.996456 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:07.996989 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:07.997005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:08.000073 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.124.254:41039: Server oryx/127.8.124.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
W20260928 05:24:08.000533 17653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:07.992360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:32827 (local address 127.8.124.254:41039)
0928 05:24:07.992514 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:07.992519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:07.992811 (+ 292us) server_negotiation.cc:407] Connection header received
0928 05:24:07.992962 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:07.992966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:07.993022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:07.993094 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:07.993859 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.994554 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.995719 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:07.996172 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:07.996270 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:08.000305 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.124.194:32827: BlockingRecv error: recv got EOF from 127.8.124.194:32827 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:24:08.374343 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17503
I20260928 05:24:08.385864 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:08.375334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.8.124.193:43777)
0928 05:24:08.375911 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:08.375918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:08.375944 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:24:08.376004 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:08.376010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:08.376250 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:08.376409 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:08.377805 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.378826 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.380066 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.380752 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.380872 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:08.383155 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:08.383163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:08.383177 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:08.383219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:08.385376 (+ 2157us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:24:08.385662 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43940: Request ticket server kudu/127.8.124.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
W20260928 05:24:08.385947 17965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43940: Request ticket server kudu/127.8.124.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:24:08.385994 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17503: Not authorized: Client connection negotiation failed: client connection to 127.8.124.193:43777: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:24:08.386201 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17503
I20260928 05:24:08.399958 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17654
I20260928 05:24:08.412349 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:08.401472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.8.124.194:33779)
0928 05:24:08.402151 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:08.402157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:08.402184 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:24:08.402249 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:08.402255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:08.402480 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:08.402638 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:08.404107 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.405194 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.406426 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.406991 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.407093 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:08.409257 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:08.409265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:08.409279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:08.409320 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:08.411783 (+ 2463us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:24:08.412087 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43530: Request ticket server kudu/127.8.124.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":369,"threads_started":1}
W20260928 05:24:08.412392 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17654: Not authorized: Client connection negotiation failed: client connection to 127.8.124.194:33779: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:24:08.412444 17966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43530: Request ticket server kudu/127.8.124.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:24:08.412530 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17654
I20260928 05:24:08.425653 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17787
I20260928 05:24:08.437557 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:08.426591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.8.124.195:43123)
0928 05:24:08.427232 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:08.427241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:08.427269 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:24:08.427339 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:08.427358 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:08.427652 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:08.427849 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:08.429091 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.430291 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.431487 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:08.432014 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:08.432108 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:08.434600 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:08.434608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:08.434621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:08.434657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:08.437067 (+ 2410us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:24:08.437315 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57862: Request ticket server kudu/127.8.124.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":311,"threads_started":1}
W20260928 05:24:08.437676 17967 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57862: Request ticket server kudu/127.8.124.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:24:08.437707 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17787: Not authorized: Client connection negotiation failed: client connection to 127.8.124.195:43123: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.124.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:24:08.437862 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17787
I20260928 05:24:08.450527 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17434
I20260928 05:24:08.451676 17496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:08.561663 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17434
2026-09-28T05:24:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20851 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T05:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:24:08Z Disabled control of system clock
I20260928 05:24:08.626981 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38917
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:24:08.928530 17972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:08.928825 17972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:08.928903 17972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:08.936146 17972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:24:08.936256 17972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:08.936292 17972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:08.936347 17972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:24:08.936395 17972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:24:08.945385 17972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:08.946679 17972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:08.948415 17972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:08.959439 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:08.959520 17977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:08.959564 17978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:08.960165 17972 server_base.cc:1034] running on GCE node
I20260928 05:24:08.960889 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:08.962267 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:08.963469 17972 hybrid_clock.cc:648] HybridClock initialized: now 1790573048963421 us; error 43 us; skew 500 ppm
I20260928 05:24:08.963790 17972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:08.965930 17972 webserver.cc:460] Webserver started at http://127.8.124.254:46663/ using document root <none> and password file <none>
I20260928 05:24:08.966567 17972 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:08.966655 17972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:08.966938 17972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:08.969651 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "5ff82544677548b78f00a8e382e6c3e2"
format_stamp: "Formatted at 2026-09-28 05:24:08 on dist-test-slave-515x"
server_key: "30f4327558234328ba8506f1cc9f1864"
server_key_iv: "0a3bf39360f68664e553d122d7d00717"
server_key_version: "encryptionkey@0"
I20260928 05:24:08.970393 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "5ff82544677548b78f00a8e382e6c3e2"
format_stamp: "Formatted at 2026-09-28 05:24:08 on dist-test-slave-515x"
server_key: "30f4327558234328ba8506f1cc9f1864"
server_key_iv: "0a3bf39360f68664e553d122d7d00717"
server_key_version: "encryptionkey@0"
I20260928 05:24:08.975127 17972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:24:08.978478 17986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:08.979715 17972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:24:08.979862 17972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "5ff82544677548b78f00a8e382e6c3e2"
format_stamp: "Formatted at 2026-09-28 05:24:08 on dist-test-slave-515x"
server_key: "30f4327558234328ba8506f1cc9f1864"
server_key_iv: "0a3bf39360f68664e553d122d7d00717"
server_key_version: "encryptionkey@0"
I20260928 05:24:08.980043 17972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:09.003041 17972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:09.003818 17972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:09.004062 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:09.020282 17972 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:38917
I20260928 05:24:09.020299 18037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:38917 every 8 connection(s)
I20260928 05:24:09.021504 17972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:24:09.022456 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 17972
I20260928 05:24:09.022635 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:24:09.023414 8691 external_mini_cluster.cc:1428] Setting key 1ade185f7209690290af2cdbe6b5324e
I20260928 05:24:09.026015 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:09.036099 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:09.025356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.8.124.254:38917)
0928 05:24:09.026273 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:09.026289 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:09.026329 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:24:09.027819 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:09.027840 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:09.028263 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:09.028589 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:09.029130 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.030430 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.032162 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.032756 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.032872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:09.034059 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:09.034066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:09.034078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:09.034592 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:09.034662 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:09.034680 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:09.034770 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:09.034971 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 05:24:09.035252 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":328,"threads_started":1}
I20260928 05:24:09.037390 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2: Bootstrap starting.
I20260928 05:24:09.040256 18038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:09.041126 18038 log.cc:824] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:09.043563 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2: No bootstrap required, opened a new log
I20260928 05:24:09.050146 18038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } }
I20260928 05:24:09.050473 18038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:09.050557 18038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff82544677548b78f00a8e382e6c3e2, State: Initialized, Role: FOLLOWER
I20260928 05:24:09.051021 18038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } }
I20260928 05:24:09.051211 18038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:24:09.051326 18038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:24:09.051466 18038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:09.053363 18038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } }
I20260928 05:24:09.053829 18038 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff82544677548b78f00a8e382e6c3e2; no voters:
I20260928 05:24:09.054401 18038 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:24:09.054546 18043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:24:09.055217 18043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff82544677548b78f00a8e382e6c3e2, State: Running, Role: LEADER
I20260928 05:24:09.055789 18043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } }
I20260928 05:24:09.056422 18038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:24:09.058372 18044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff82544677548b78f00a8e382e6c3e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } } }
I20260928 05:24:09.058727 18044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:09.058663 18045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff82544677548b78f00a8e382e6c3e2. Latest consensus state: current_term: 1 leader_uuid: "5ff82544677548b78f00a8e382e6c3e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff82544677548b78f00a8e382e6c3e2" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38917 } } }
I20260928 05:24:09.060070 18045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2 [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:09.062372 18051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:24:09.066975 18051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:24:09.075016 18051 catalog_manager.cc:1347] Generated new cluster ID: 490f0facb05d43c5875ac8db46db9b4c
I20260928 05:24:09.075104 18051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:24:09.087218 18051 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:24:09.088373 18051 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:24:09.093376 18051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff82544677548b78f00a8e382e6c3e2: Generated new TSK 0
I20260928 05:24:09.094135 18051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:24:09.107088 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--builtin_ntp_servers=127.8.124.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 05:24:09.538408 18062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:09.538674 18062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:09.538755 18062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:09.545951 18062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:09.546037 18062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:09.546144 18062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:24:09.555173 18062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:09.556463 18062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:09.558338 18062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:09.570569 18067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:09.570571 18068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:09.570705 18070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:09.571236 18062 server_base.cc:1034] running on GCE node
I20260928 05:24:09.571789 18062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:09.572683 18062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:09.573864 18062 hybrid_clock.cc:648] HybridClock initialized: now 1790573049573805 us; error 56 us; skew 500 ppm
I20260928 05:24:09.574162 18062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:09.576284 18062 webserver.cc:460] Webserver started at http://127.8.124.193:36749/ using document root <none> and password file <none>
I20260928 05:24:09.576989 18062 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:09.577080 18062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:09.577378 18062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:09.580157 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "fd5e53617c464a4e80324021db4c1e46"
format_stamp: "Formatted at 2026-09-28 05:24:09 on dist-test-slave-515x"
server_key: "e86ed6d86c0b819db696f8c88bcf9c89"
server_key_iv: "851ca1c0c3241420a90df6886789f2cd"
server_key_version: "encryptionkey@0"
I20260928 05:24:09.580896 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "fd5e53617c464a4e80324021db4c1e46"
format_stamp: "Formatted at 2026-09-28 05:24:09 on dist-test-slave-515x"
server_key: "e86ed6d86c0b819db696f8c88bcf9c89"
server_key_iv: "851ca1c0c3241420a90df6886789f2cd"
server_key_version: "encryptionkey@0"
I20260928 05:24:09.585901 18062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:24:09.589169 18076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:09.590617 18062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:24:09.590768 18062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "fd5e53617c464a4e80324021db4c1e46"
format_stamp: "Formatted at 2026-09-28 05:24:09 on dist-test-slave-515x"
server_key: "e86ed6d86c0b819db696f8c88bcf9c89"
server_key_iv: "851ca1c0c3241420a90df6886789f2cd"
server_key_version: "encryptionkey@0"
I20260928 05:24:09.590966 18062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:09.615068 18062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:09.615907 18062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:09.616185 18062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:09.617130 18062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:09.618546 18062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:09.618637 18062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:09.618741 18062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:09.618808 18062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:09.646579 18062 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:36919
I20260928 05:24:09.646641 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:36919 every 8 connection(s)
I20260928 05:24:09.647805 18062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:24:09.657447 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18062
I20260928 05:24:09.657667 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:24:09.658337 8691 external_mini_cluster.cc:1428] Setting key c244fcf24621abb79cbcd2e2a1e5b6a3
I20260928 05:24:09.659466 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:09.650163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:51053 (local address 127.8.124.254:38917)
0928 05:24:09.650556 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:09.650560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:09.651738 (+ 1178us) server_negotiation.cc:407] Connection header received
0928 05:24:09.653373 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:09.653377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:09.653440 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:09.653524 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:09.654507 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.655196 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.656666 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.657245 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.657362 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:09.658668 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:09.658675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:09.658680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:09.658699 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:09.658743 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:09.658758 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:09.658847 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:09.659045 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 05:24:09.659120 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":192,"threads_started":1}
I20260928 05:24:09.660532 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:09.650753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38917 (local address 127.8.124.193:51053)
0928 05:24:09.651580 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:09.651632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:09.653138 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:09.653746 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:09.653754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:09.653844 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:09.654374 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:09.654389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.655335 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.655339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:09.656529 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:09.656540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:09.657439 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:09.657444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:09.657687 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:09.658355 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:09.658358 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:09.658394 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:09.658484 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:09.658842 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:09.658848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:09.658853 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:24:09.659021 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 05:24:09.659337 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":239,"threads_started":1}
I20260928 05:24:09.661108 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--builtin_ntp_servers=127.8.124.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:24:09.661993 18189 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38917
I20260928 05:24:09.662276 18189 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:09.662992 18189 heartbeater.cc:511] Master 127.8.124.254:38917 requested a full tablet report, sending...
I20260928 05:24:09.664860 18003 ts_manager.cc:194] Registered new tserver with Master: fd5e53617c464a4e80324021db4c1e46 (127.8.124.193:36919)
I20260928 05:24:09.666296 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.193:51053
W20260928 05:24:09.967765 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:09.968024 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:09.968125 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:09.975186 18193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:09.975281 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:09.975425 18193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:24:09.984422 18193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:09.985687 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:09.987450 18193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:09.999503 18198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:09.999482 18199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:09.999617 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:10.000306 18193 server_base.cc:1034] running on GCE node
I20260928 05:24:10.000835 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:10.001752 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:10.002979 18193 hybrid_clock.cc:648] HybridClock initialized: now 1790573050002932 us; error 35 us; skew 500 ppm
I20260928 05:24:10.003291 18193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:10.009918 18193 webserver.cc:460] Webserver started at http://127.8.124.194:40197/ using document root <none> and password file <none>
I20260928 05:24:10.010648 18193 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:10.010746 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:10.011049 18193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:10.013921 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "66c3025eb5d2449aa9432d75acfb7f52"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "e3b9a6d6dad686bc4f879717dc91d2f4"
server_key_iv: "da88402ed2ef51665a82529b210dc0b3"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.014667 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "66c3025eb5d2449aa9432d75acfb7f52"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "e3b9a6d6dad686bc4f879717dc91d2f4"
server_key_iv: "da88402ed2ef51665a82529b210dc0b3"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.019613 18193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:24:10.023860 18207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.025414 18193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:24:10.025568 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "66c3025eb5d2449aa9432d75acfb7f52"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "e3b9a6d6dad686bc4f879717dc91d2f4"
server_key_iv: "da88402ed2ef51665a82529b210dc0b3"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.025825 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:10.056625 18193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:10.057433 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:10.057737 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:10.058620 18193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:10.059980 18193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:10.060068 18193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.060143 18193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:10.060194 18193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.088215 18193 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45123
I20260928 05:24:10.088284 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45123 every 8 connection(s)
I20260928 05:24:10.089463 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:24:10.090009 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18193
I20260928 05:24:10.090144 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:24:10.090700 8691 external_mini_cluster.cc:1428] Setting key c9938cfcf0fcac9665adbd3df6bbf8de
I20260928 05:24:10.093626 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--builtin_ntp_servers=127.8.124.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:24:10.102901 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.091973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:55307 (local address 127.8.124.254:38917)
0928 05:24:10.092148 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.092154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.093778 (+ 1624us) server_negotiation.cc:407] Connection header received
0928 05:24:10.096068 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.096072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.096133 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.096210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.097433 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.098153 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.100201 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.100735 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.100848 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.102185 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:10.102198 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:10.102203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:10.102222 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:10.102257 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:10.102271 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:10.102353 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:10.102512 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 05:24:10.102602 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:24:10.103808 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.092566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38917 (local address 127.8.124.194:55307)
0928 05:24:10.093596 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:10.093684 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:10.095804 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:10.096426 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:10.096436 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:10.096549 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:10.097256 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:10.097279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.098282 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.098287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:10.100055 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:10.100069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.100949 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.100956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:10.101195 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.101879 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:10.101882 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:10.101919 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:10.101995 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:10.102350 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:10.102356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:10.102361 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:24:10.102487 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 05:24:10.102773 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
I20260928 05:24:10.105084 18320 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38917
I20260928 05:24:10.105392 18320 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:10.106244 18320 heartbeater.cc:511] Master 127.8.124.254:38917 requested a full tablet report, sending...
I20260928 05:24:10.107941 18003 ts_manager.cc:194] Registered new tserver with Master: 66c3025eb5d2449aa9432d75acfb7f52 (127.8.124.194:45123)
I20260928 05:24:10.108701 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.194:55307
W20260928 05:24:10.405488 18323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:10.405795 18323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:10.405927 18323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:10.413756 18323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:10.413851 18323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:10.413988 18323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:24:10.422837 18323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:10.424026 18323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:10.425784 18323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:10.437664 18331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:10.437633 18328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:10.437696 18329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:10.438349 18323 server_base.cc:1034] running on GCE node
I20260928 05:24:10.438880 18323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:10.439670 18323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:10.440855 18323 hybrid_clock.cc:648] HybridClock initialized: now 1790573050440812 us; error 33 us; skew 500 ppm
I20260928 05:24:10.441147 18323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:10.443251 18323 webserver.cc:460] Webserver started at http://127.8.124.195:45331/ using document root <none> and password file <none>
I20260928 05:24:10.443866 18323 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:10.443949 18323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:10.444232 18323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:10.446818 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "e45f62c3db87443080fa360649d0925d"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "7ef9f9688c53fd21992e032f29834b2f"
server_key_iv: "32806e6ebf675feb98f3d768bc7283cf"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.447523 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "e45f62c3db87443080fa360649d0925d"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "7ef9f9688c53fd21992e032f29834b2f"
server_key_iv: "32806e6ebf675feb98f3d768bc7283cf"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.452077 18323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:24:10.455580 18337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.456912 18323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:24:10.457053 18323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "e45f62c3db87443080fa360649d0925d"
format_stamp: "Formatted at 2026-09-28 05:24:10 on dist-test-slave-515x"
server_key: "7ef9f9688c53fd21992e032f29834b2f"
server_key_iv: "32806e6ebf675feb98f3d768bc7283cf"
server_key_version: "encryptionkey@0"
I20260928 05:24:10.457234 18323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:10.486804 18323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:10.487624 18323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:10.487887 18323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:10.488781 18323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:10.490110 18323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:10.490202 18323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.490306 18323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:10.490391 18323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:10.520742 18323 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:44971
I20260928 05:24:10.520802 18449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:44971 every 8 connection(s)
I20260928 05:24:10.521960 18323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:24:10.526780 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18323
I20260928 05:24:10.526904 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:24:10.527344 8691 external_mini_cluster.cc:1428] Setting key 54d3d342a679d70bb304290503a96105
I20260928 05:24:10.533520 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.524205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:54949 (local address 127.8.124.254:38917)
0928 05:24:10.524370 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.524376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.525639 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 05:24:10.527394 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.527399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.527471 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.527553 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.528692 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.529537 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.531093 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.531493 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.531597 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.532756 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:10.532760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:10.532763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:10.532779 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:10.532808 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:10.532819 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:10.532897 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:10.533096 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 05:24:10.533184 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:24:10.534497 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.524644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38917 (local address 127.8.124.195:54949)
0928 05:24:10.525470 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:10.525505 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:10.527140 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:10.527776 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:10.527784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:10.527874 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:10.528549 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:10.528567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.529688 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.529693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:10.530968 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:10.530983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.531685 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.531692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:10.531909 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.532484 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:10.532487 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:10.532523 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:10.532591 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:10.532911 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:10.532921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:10.532929 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:24:10.533077 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 05:24:10.533420 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":250,"threads_started":1}
I20260928 05:24:10.535588 18450 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38917
I20260928 05:24:10.535838 18450 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:10.536526 18450 heartbeater.cc:511] Master 127.8.124.254:38917 requested a full tablet report, sending...
I20260928 05:24:10.537797 18003 ts_manager.cc:194] Registered new tserver with Master: e45f62c3db87443080fa360649d0925d (127.8.124.195:44971)
I20260928 05:24:10.538537 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.195:54949
I20260928 05:24:10.541921 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:24:10.547610 18191 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:10.546183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.8.124.254:38917)
0928 05:24:10.546312 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.546316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.546812 (+ 496us) server_negotiation.cc:407] Connection header received
0928 05:24:10.546924 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.546927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.547002 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.547074 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.547411 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40016: BlockingRecv error: recv got EOF from 127.0.0.1:40016 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 05:24:10.548337 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18062
I20260928 05:24:10.555006 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.549111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.8.124.193:36919)
0928 05:24:10.549561 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.549567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.549589 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:24:10.549691 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.549697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.549880 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.550005 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.550314 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.551258 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.552404 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.552920 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.553011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.553875 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:10.553883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:10.553895 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:10.554403 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:10.554443 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:10.554461 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:10.554552 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:10.554639 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 05:24:10.554745 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":197,"threads_started":1}
I20260928 05:24:10.556090 18184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:10.634344 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18062
I20260928 05:24:10.647059 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18193
I20260928 05:24:10.654152 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.647856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.8.124.194:45123)
0928 05:24:10.648377 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.648384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.648412 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:24:10.648470 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.648476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.648686 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.648813 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.649181 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.650229 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.651362 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.651899 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.651993 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.652882 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:10.652890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:10.652903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:10.653463 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:10.653504 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:10.653522 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:10.653640 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:10.653743 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 05:24:10.653862 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260928 05:24:10.655318 18315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:10.733177 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18193
I20260928 05:24:10.745682 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18323
I20260928 05:24:10.753224 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:10.746627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.8.124.195:44971)
0928 05:24:10.747293 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:10.747306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:10.747336 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:24:10.747396 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:10.747402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:10.747611 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:10.747737 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:10.748152 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.749141 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.750318 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:10.750857 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:10.750950 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:10.751847 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:10.751854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:10.751868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:10.752488 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:10.752534 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:10.752562 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:10.752645 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:10.752762 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 05:24:10.752876 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":336,"threads_started":1}
I20260928 05:24:10.754383 18445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:10.834247 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18323
I20260928 05:24:10.846514 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 17972
I20260928 05:24:10.847620 18033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:10.945524 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 17972
2026-09-28T05:24:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2379 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T05:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:24:10Z Disabled control of system clock
I20260928 05:24:11.006584 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:39649
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:24:11.308379 18469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:11.308645 18469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:11.308760 18469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:11.315598 18469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:24:11.315692 18469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:11.315752 18469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:11.315822 18469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:24:11.315877 18469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:24:11.324537 18469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:39649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:11.325764 18469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:11.327438 18469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:11.338025 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:11.338085 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:11.338109 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:11.339198 18469 server_base.cc:1034] running on GCE node
I20260928 05:24:11.339936 18469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:11.341199 18469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:11.342381 18469 hybrid_clock.cc:648] HybridClock initialized: now 1790573051342330 us; error 47 us; skew 500 ppm
I20260928 05:24:11.342681 18469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:11.344724 18469 webserver.cc:460] Webserver started at http://127.8.124.254:37629/ using document root <none> and password file <none>
I20260928 05:24:11.345374 18469 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:11.345466 18469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:11.345793 18469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:11.348490 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "72afd90f17c844778b7c334c2e2d320d"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fec8cc53276a1ead873eaaeae68860c4"
server_key_iv: "e09b7cf10568f7591450d8c95d598b5f"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.349195 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "72afd90f17c844778b7c334c2e2d320d"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fec8cc53276a1ead873eaaeae68860c4"
server_key_iv: "e09b7cf10568f7591450d8c95d598b5f"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.353884 18469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260928 05:24:11.356945 18483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:11.358166 18469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:24:11.358302 18469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "72afd90f17c844778b7c334c2e2d320d"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fec8cc53276a1ead873eaaeae68860c4"
server_key_iv: "e09b7cf10568f7591450d8c95d598b5f"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.358486 18469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:11.363646 18469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:11.364324 18469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:11.364588 18469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:11.380436 18469 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:39649
I20260928 05:24:11.380453 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:39649 every 8 connection(s)
I20260928 05:24:11.381821 18469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:24:11.383363 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18469
I20260928 05:24:11.383561 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:24:11.384153 8691 external_mini_cluster.cc:1428] Setting key d4e2e6790d403487ad1480c0cca24aee
I20260928 05:24:11.385680 18535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:11.392788 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:11.386239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.8.124.254:39649)
0928 05:24:11.387250 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:11.387263 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:11.387309 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:24:11.388887 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:11.388914 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:11.389452 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:11.389894 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:11.390870 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:11.390882 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:11.390899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:11.390915 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:11.391017 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:11.391038 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:11.391262 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:11.391382 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 05:24:11.391637 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":313,"threads_started":1}
I20260928 05:24:11.398247 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d: Bootstrap starting.
I20260928 05:24:11.401116 18535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:11.402165 18535 log.cc:824] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:11.404572 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d: No bootstrap required, opened a new log
I20260928 05:24:11.412397 18535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } }
I20260928 05:24:11.412779 18535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:11.412873 18535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72afd90f17c844778b7c334c2e2d320d, State: Initialized, Role: FOLLOWER
I20260928 05:24:11.413446 18535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } }
I20260928 05:24:11.413708 18535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:24:11.413831 18535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:24:11.413975 18535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:11.415940 18535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } }
I20260928 05:24:11.416389 18535 leader_election.cc:302] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72afd90f17c844778b7c334c2e2d320d; no voters:
I20260928 05:24:11.416898 18535 leader_election.cc:288] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:24:11.417025 18540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:24:11.417658 18540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [term 1 LEADER]: Becoming Leader. State: Replica: 72afd90f17c844778b7c334c2e2d320d, State: Running, Role: LEADER
I20260928 05:24:11.418160 18540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } }
I20260928 05:24:11.418854 18535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:24:11.420707 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72afd90f17c844778b7c334c2e2d320d. Latest consensus state: current_term: 1 leader_uuid: "72afd90f17c844778b7c334c2e2d320d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } } }
I20260928 05:24:11.421060 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:11.424798 18547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:24:11.424680 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72afd90f17c844778b7c334c2e2d320d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72afd90f17c844778b7c334c2e2d320d" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 39649 } } }
I20260928 05:24:11.424965 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:11.429713 18547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:24:11.437810 18547 catalog_manager.cc:1347] Generated new cluster ID: 17f6e3970bd3493280a8582fe0663725
I20260928 05:24:11.437906 18547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:24:11.464448 18547 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:24:11.464824 18547 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:24:11.474498 18547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72afd90f17c844778b7c334c2e2d320d: Generated new TSK 0
I20260928 05:24:11.475247 18547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:24:11.488072 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--builtin_ntp_servers=127.8.124.212:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:24:11.813009 18559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:11.813259 18559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:11.813351 18559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:11.820411 18559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:11.820501 18559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:11.820613 18559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:24:11.829145 18559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:11.830327 18559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:11.831988 18559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:11.844121 18565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:11.844153 18567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:11.844375 18564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:11.845300 18559 server_base.cc:1034] running on GCE node
I20260928 05:24:11.845974 18559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:11.846784 18559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:11.847976 18559 hybrid_clock.cc:648] HybridClock initialized: now 1790573051847921 us; error 55 us; skew 500 ppm
I20260928 05:24:11.848237 18559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:11.850212 18559 webserver.cc:460] Webserver started at http://127.8.124.193:39105/ using document root <none> and password file <none>
I20260928 05:24:11.850811 18559 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:11.850890 18559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:11.851140 18559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:11.855418 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "a5b4793cf9a44d42beb5be9bbbddd940"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fa79b74cb04290112bccf440588bdeb7"
server_key_iv: "89b156dfe15d07d1ac328ec2d2575343"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.856112 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b4793cf9a44d42beb5be9bbbddd940"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fa79b74cb04290112bccf440588bdeb7"
server_key_iv: "89b156dfe15d07d1ac328ec2d2575343"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.860860 18559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:24:11.864001 18573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:11.865198 18559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:24:11.865370 18559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "a5b4793cf9a44d42beb5be9bbbddd940"
format_stamp: "Formatted at 2026-09-28 05:24:11 on dist-test-slave-515x"
server_key: "fa79b74cb04290112bccf440588bdeb7"
server_key_iv: "89b156dfe15d07d1ac328ec2d2575343"
server_key_version: "encryptionkey@0"
I20260928 05:24:11.865540 18559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:11.870682 18559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:11.871382 18559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:11.871601 18559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:11.872398 18559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:11.873668 18559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:11.873752 18559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:11.873852 18559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:11.873925 18559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:11.902104 18559 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:45891
I20260928 05:24:11.902148 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:45891 every 8 connection(s)
I20260928 05:24:11.903283 18559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:24:11.911005 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18559
I20260928 05:24:11.911026 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:11.905573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:47227 (local address 127.8.124.254:39649)
0928 05:24:11.905987 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:11.905991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:11.907143 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 05:24:11.908951 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:11.908957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:11.909013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:11.909108 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:11.910308 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:11.910314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:11.910319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:11.910344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:11.910394 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:11.910408 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:11.910507 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:11.910680 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 05:24:11.910741 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":190,"threads_started":1}
I20260928 05:24:11.911190 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:24:11.911770 8691 external_mini_cluster.cc:1428] Setting key d0539d669a68ba3b01e6de6a72a1f49d
I20260928 05:24:11.911887 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:11.906110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:39649 (local address 127.8.124.193:47227)
0928 05:24:11.906991 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:11.907032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:11.908666 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:11.909327 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:11.909334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:11.909431 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:11.910038 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:11.910040 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:11.910075 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:11.910153 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:11.910496 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:11.910505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:11.910512 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:24:11.910741 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 05:24:11.910977 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260928 05:24:11.913214 18686 heartbeater.cc:347] Connected to a master server at 127.8.124.254:39649
I20260928 05:24:11.913478 18686 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:11.914106 18686 heartbeater.cc:511] Master 127.8.124.254:39649 requested a full tablet report, sending...
I20260928 05:24:11.914710 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--builtin_ntp_servers=127.8.124.212:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 05:24:11.916018 18499 ts_manager.cc:194] Registered new tserver with Master: a5b4793cf9a44d42beb5be9bbbddd940 (127.8.124.193:45891)
W20260928 05:24:12.222244 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:12.222517 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:12.222651 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:12.229840 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:12.229934 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:12.230089 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:24:12.238756 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:12.239949 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:12.241626 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:12.253404 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:12.253376 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:12.253346 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:12.254004 18690 server_base.cc:1034] running on GCE node
I20260928 05:24:12.254490 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:12.255285 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:12.256460 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790573052256412 us; error 44 us; skew 500 ppm
I20260928 05:24:12.256716 18690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:12.262868 18690 webserver.cc:460] Webserver started at http://127.8.124.194:37299/ using document root <none> and password file <none>
I20260928 05:24:12.263571 18690 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:12.263650 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:12.263907 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:12.266568 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "43d6ab28b0ea43ab8cc27593cfd99789"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "c7e52fdf27d25580f10f7690250cc04a"
server_key_iv: "7a0a4c4a641ac7bb0960c9b5e3ffd0c0"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.267242 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "43d6ab28b0ea43ab8cc27593cfd99789"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "c7e52fdf27d25580f10f7690250cc04a"
server_key_iv: "7a0a4c4a641ac7bb0960c9b5e3ffd0c0"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.272078 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:24:12.275202 18704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.276381 18690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:24:12.276513 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "43d6ab28b0ea43ab8cc27593cfd99789"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "c7e52fdf27d25580f10f7690250cc04a"
server_key_iv: "7a0a4c4a641ac7bb0960c9b5e3ffd0c0"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.276700 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:12.281858 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:12.282588 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:12.282804 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:12.283634 18690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:12.284916 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:12.284996 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.285094 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:12.285157 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.312988 18690 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:36959
I20260928 05:24:12.313051 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:36959 every 8 connection(s)
I20260928 05:24:12.314193 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:24:12.321548 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.316501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:58355 (local address 127.8.124.254:39649)
0928 05:24:12.316680 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.316686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.318058 (+ 1372us) server_negotiation.cc:407] Connection header received
0928 05:24:12.319671 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.319675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.319740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.319824 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.320957 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:12.320962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:12.320965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:12.320981 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:12.321011 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:12.321022 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:12.321109 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:12.321283 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 05:24:12.321340 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:24:12.322398 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.317043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:39649 (local address 127.8.124.194:58355)
0928 05:24:12.317919 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:12.317956 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:12.319426 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:12.320014 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:12.320024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:12.320122 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:12.320694 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:12.320696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:12.320732 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:12.320810 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:12.321089 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:12.321095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:12.321100 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:24:12.321289 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 05:24:12.321476 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":239,"threads_started":1}
I20260928 05:24:12.323616 18817 heartbeater.cc:347] Connected to a master server at 127.8.124.254:39649
I20260928 05:24:12.323879 18817 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:12.324424 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18690
I20260928 05:24:12.324445 18817 heartbeater.cc:511] Master 127.8.124.254:39649 requested a full tablet report, sending...
I20260928 05:24:12.324579 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:24:12.324957 8691 external_mini_cluster.cc:1428] Setting key edcf05f50df87faadb255cba0f26ea60
I20260928 05:24:12.325713 18499 ts_manager.cc:194] Registered new tserver with Master: 43d6ab28b0ea43ab8cc27593cfd99789 (127.8.124.194:36959)
I20260928 05:24:12.327275 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--builtin_ntp_servers=127.8.124.212:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:24:12.640369 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:12.640641 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:12.640776 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:12.647856 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:12.647960 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:12.648108 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:24:12.656878 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:12.658197 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:12.659876 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:12.671916 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:12.672005 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:12.671999 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:12.672948 18820 server_base.cc:1034] running on GCE node
I20260928 05:24:12.673480 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:12.674383 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:12.675585 18820 hybrid_clock.cc:648] HybridClock initialized: now 1790573052675526 us; error 50 us; skew 500 ppm
I20260928 05:24:12.675879 18820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:24:12.678054 18820 webserver.cc:460] Webserver started at http://127.8.124.195:46175/ using document root <none> and password file <none>
I20260928 05:24:12.678666 18820 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:12.678762 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:12.679061 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:12.681806 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "a119a6305ca242b6b085b10d759cf84b"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "d1eb61f27a3347c871042eb33426ca37"
server_key_iv: "45862dba0dd21a0c16a99eb63ca6dbbb"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.682520 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "a119a6305ca242b6b085b10d759cf84b"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "d1eb61f27a3347c871042eb33426ca37"
server_key_iv: "45862dba0dd21a0c16a99eb63ca6dbbb"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.687237 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:24:12.690409 18834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.691591 18820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260928 05:24:12.691731 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "a119a6305ca242b6b085b10d759cf84b"
format_stamp: "Formatted at 2026-09-28 05:24:12 on dist-test-slave-515x"
server_key: "d1eb61f27a3347c871042eb33426ca37"
server_key_iv: "45862dba0dd21a0c16a99eb63ca6dbbb"
server_key_version: "encryptionkey@0"
I20260928 05:24:12.691931 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:12.697273 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:12.698045 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:12.698275 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:12.699084 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:12.700342 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:12.700434 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.700536 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:12.700600 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:12.728821 18820 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:34519
I20260928 05:24:12.728875 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:34519 every 8 connection(s)
I20260928 05:24:12.730079 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:24:12.735399 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18820
I20260928 05:24:12.735551 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:24:12.735935 8691 external_mini_cluster.cc:1428] Setting key fbc14bd850196de25b2e04991e0ce01d
I20260928 05:24:12.737586 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.732364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:47251 (local address 127.8.124.254:39649)
0928 05:24:12.732519 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.732524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.733861 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 05:24:12.735442 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.735448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.735514 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.735606 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.736830 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:12.736834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:12.736837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:12.736851 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:12.736881 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:12.736894 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:12.737006 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:12.737220 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 05:24:12.737282 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:24:12.738525 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.732838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:39649 (local address 127.8.124.195:47251)
0928 05:24:12.733723 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:12.733758 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:12.735173 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:12.735821 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:12.735830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:12.735931 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:12.736559 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:24:12.736562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:12.736600 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:24:12.736685 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:12.737002 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:12.737011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:12.737019 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:24:12.737283 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 05:24:12.737536 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":243,"threads_started":1}
I20260928 05:24:12.739760 18947 heartbeater.cc:347] Connected to a master server at 127.8.124.254:39649
I20260928 05:24:12.739995 18947 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:12.740516 18947 heartbeater.cc:511] Master 127.8.124.254:39649 requested a full tablet report, sending...
I20260928 05:24:12.741719 18500 ts_manager.cc:194] Registered new tserver with Master: a119a6305ca242b6b085b10d759cf84b (127.8.124.195:34519)
I20260928 05:24:12.750798 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:24:12.756422 18688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:24:12.755115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.8.124.254:39649)
0928 05:24:12.755290 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.755296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.755692 (+ 396us) server_negotiation.cc:407] Connection header received
0928 05:24:12.755828 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.755831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.755879 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.755955 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.756182 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40912: BlockingRecv error: recv got EOF from 127.0.0.1:40912 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:24:12.756852 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18559
I20260928 05:24:12.760116 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.757706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.8.124.193:45891)
0928 05:24:12.758170 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.758177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.758199 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:24:12.758249 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.758255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.758450 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.758578 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.759464 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:12.759470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:12.759481 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:12.759496 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:12.759532 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:12.759547 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:12.759656 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:12.759777 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 05:24:12.759869 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":191,"threads_started":1}
I20260928 05:24:12.761248 18681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:12.838778 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18559
I20260928 05:24:12.851408 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18690
I20260928 05:24:12.854746 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.852220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.8.124.194:36959)
0928 05:24:12.852674 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.852681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.852706 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:24:12.852766 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.852772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.852980 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.853115 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.854070 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:12.854076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:12.854088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:12.854105 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:12.854143 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:12.854156 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:12.854271 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:12.854397 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:24:12.854504 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":212,"threads_started":1}
I20260928 05:24:12.855901 18812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:12.934240 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18690
I20260928 05:24:12.946581 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18820
I20260928 05:24:12.950347 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:12.947434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.8.124.195:34519)
0928 05:24:12.948029 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:12.948036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:12.948066 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:24:12.948126 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:12.948132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:12.948353 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:12.948544 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:12.949629 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:12.949637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:12.949653 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:24:12.949675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:12.949717 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:24:12.949733 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:12.949885 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:12.949955 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 05:24:12.950052 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":291,"threads_started":1}
I20260928 05:24:12.951488 18942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:13.029050 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18820
I20260928 05:24:13.043015 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18469
I20260928 05:24:13.044122 18530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:13.141248 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18469
2026-09-28T05:24:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2195 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:24:13 dist-test-slave-515x krb5kdc[18965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:24:13 dist-test-slave-515x krb5kdc[18965](info): set up 2 sockets
Sep 28 05:24:13 dist-test-slave-515x krb5kdc[18965](info): commencing operation
krb5kdc: starting...
W20260928 05:24:14.146195 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:24:14 dist-test-slave-515x krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:24:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:24:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:24:14.345541 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43889
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:24:14.655150 18978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:14.655436 18978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:14.655571 18978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:14.662688 18978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:24:14.662793 18978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:14.662848 18978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:14.662927 18978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:24:14.662984 18978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:24:14.672714 18978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:14.674077 18978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:14.675390 18978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:14.686332 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:14.686406 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:14.686374 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:14.687111 18978 server_base.cc:1034] running on GCE node
I20260928 05:24:14.687834 18978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:14.689181 18978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:14.690378 18978 hybrid_clock.cc:648] HybridClock initialized: now 1790573054690330 us; error 38 us; skew 500 ppm
I20260928 05:24:14.690693 18978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:24:14 dist-test-slave-515x krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:24:14.693616 18978 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:24:14.695039 18978 webserver.cc:460] Webserver started at http://127.8.124.254:40495/ using document root <none> and password file <none>
I20260928 05:24:14.695668 18978 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:14.695772 18978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:14.696054 18978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:14.698866 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "cf79a364b09e4f85a36dff5a6080c6f3"
format_stamp: "Formatted at 2026-09-28 05:24:14 on dist-test-slave-515x"
server_key: "5083ad093d272e7de6999c8632b94284"
server_key_iv: "32e0c5a2cbfaf568ce24585abcd16333"
server_key_version: "encryptionkey@0"
I20260928 05:24:14.699561 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "cf79a364b09e4f85a36dff5a6080c6f3"
format_stamp: "Formatted at 2026-09-28 05:24:14 on dist-test-slave-515x"
server_key: "5083ad093d272e7de6999c8632b94284"
server_key_iv: "32e0c5a2cbfaf568ce24585abcd16333"
server_key_version: "encryptionkey@0"
I20260928 05:24:14.704274 18978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 05:24:14.707392 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:14.708602 18978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:24:14.708751 18978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "cf79a364b09e4f85a36dff5a6080c6f3"
format_stamp: "Formatted at 2026-09-28 05:24:14 on dist-test-slave-515x"
server_key: "5083ad093d272e7de6999c8632b94284"
server_key_iv: "32e0c5a2cbfaf568ce24585abcd16333"
server_key_version: "encryptionkey@0"
I20260928 05:24:14.708925 18978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:14.728662 18978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:14.732735 18978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:14.732991 18978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:14.748993 18978 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:43889
I20260928 05:24:14.749015 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:43889 every 8 connection(s)
I20260928 05:24:14.750311 18978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:24:14.753831 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:14.754570 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 18978
I20260928 05:24:14.754758 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:24:14.755337 8691 external_mini_cluster.cc:1428] Setting key 7aa98723170d0457ccb3b6ac189368ae
I20260928 05:24:14.767002 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: Bootstrap starting.
Sep 28 05:24:14 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:24:14.770582 19045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:14.771791 19045 log.cc:824] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:14.775081 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: No bootstrap required, opened a new log
I20260928 05:24:14.778944 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:14.757234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.8.124.254:43889)
0928 05:24:14.757994 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:14.758009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:14.758069 (+ 60us) server_negotiation.cc:407] Connection header received
0928 05:24:14.759794 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:14.759821 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:14.760335 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:14.760765 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:14.762115 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:14.764061 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:14.765351 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:14.766363 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:14.766531 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:14.770941 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:14.770955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:14.770976 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:14.771050 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:14.774649 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:14.775328 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:14.775336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:14.775344 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:14.775556 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:14.775903 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:14.775908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:14.775911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:14.776557 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:14.776853 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:14.777188 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 05:24:14.777637 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260928 05:24:14.783574 19045 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } }
I20260928 05:24:14.783895 19045 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:14.783982 19045 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf79a364b09e4f85a36dff5a6080c6f3, State: Initialized, Role: FOLLOWER
I20260928 05:24:14.784502 19045 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } }
I20260928 05:24:14.784703 19045 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:24:14.784793 19045 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:24:14.784914 19045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:14.787225 19045 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } }
I20260928 05:24:14.787657 19045 leader_election.cc:302] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf79a364b09e4f85a36dff5a6080c6f3; no voters:
I20260928 05:24:14.788161 19045 leader_election.cc:288] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:24:14.788282 19050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:24:14.789211 19050 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [term 1 LEADER]: Becoming Leader. State: Replica: cf79a364b09e4f85a36dff5a6080c6f3, State: Running, Role: LEADER
I20260928 05:24:14.789737 19050 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } }
I20260928 05:24:14.790169 19045 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:24:14.792289 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } } }
I20260928 05:24:14.792244 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf79a364b09e4f85a36dff5a6080c6f3. Latest consensus state: current_term: 1 leader_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79a364b09e4f85a36dff5a6080c6f3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43889 } } }
I20260928 05:24:14.792598 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:14.792603 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:24:14.793318 19058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:24:14.798236 19058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:24:14.807001 19058 catalog_manager.cc:1347] Generated new cluster ID: 3cddb77f9d5343f689d8026d535aefb0
I20260928 05:24:14.807122 19058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:24:14.844910 19058 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:24:14.846714 19058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:24:14.862075 19058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: Generated new TSK 0
I20260928 05:24:14.863095 19058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:24:14.920717 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:24:15.234694 19071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:15.234983 19071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:15.235119 19071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:15.242198 19071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:15.242308 19071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:15.242455 19071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:24:15.251744 19071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:15.252923 19071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:15.254158 19071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:15.266006 19076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:15.266062 19079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:15.266078 19077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:15.266705 19071 server_base.cc:1034] running on GCE node
I20260928 05:24:15.267227 19071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:15.268086 19071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:15.269273 19071 hybrid_clock.cc:648] HybridClock initialized: now 1790573055269230 us; error 37 us; skew 500 ppm
I20260928 05:24:15.269552 19071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:24:15 dist-test-slave-515x krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:24:15.275596 19071 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:24:15.276829 19071 webserver.cc:460] Webserver started at http://127.8.124.193:32889/ using document root <none> and password file <none>
I20260928 05:24:15.277475 19071 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:15.277628 19071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:15.277968 19071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:15.280812 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "eedbd051c7bc4847a5c6af6cc16488cb"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "10f72a6d243cdec3236473ba31159e5f"
server_key_iv: "9666df5f26e4a5e9c133fbb092cd7eda"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.281481 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "eedbd051c7bc4847a5c6af6cc16488cb"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "10f72a6d243cdec3236473ba31159e5f"
server_key_iv: "9666df5f26e4a5e9c133fbb092cd7eda"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.286114 19071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 05:24:15.289214 19086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.290599 19071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:24:15.290742 19071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "eedbd051c7bc4847a5c6af6cc16488cb"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "10f72a6d243cdec3236473ba31159e5f"
server_key_iv: "9666df5f26e4a5e9c133fbb092cd7eda"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.290936 19071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:15.308543 19071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:15.312423 19071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:15.312657 19071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:15.313498 19071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:15.314693 19071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:15.314765 19071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.314879 19071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:15.314936 19071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.342664 19071 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:42127
I20260928 05:24:15.342713 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:42127 every 8 connection(s)
I20260928 05:24:15.343863 19071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:24:15.353251 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 19071
I20260928 05:24:15.353483 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:24:15.354213 8691 external_mini_cluster.cc:1428] Setting key 3add00470e16f4e9094e59901b3fb475
Sep 28 05:24:15 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:24:15.362986 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:15.346232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:49101 (local address 127.8.124.254:43889)
0928 05:24:15.346647 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:15.346652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:15.347599 (+ 947us) server_negotiation.cc:407] Connection header received
0928 05:24:15.349255 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:15.349259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:15.349323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:15.349456 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:15.350869 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.351634 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.353066 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.353657 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.353784 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:15.357170 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:15.357176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:15.357181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:15.357216 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:15.360000 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:15.360716 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:15.360720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:15.360723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:15.360846 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:15.361244 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:15.361248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:15.361251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:15.361568 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:15.361685 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:15.362460 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 05:24:15.362658 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":202,"threads_started":1}
I20260928 05:24:15.364017 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:15.346773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43889 (local address 127.8.124.193:49101)
0928 05:24:15.347437 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:15.347474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:15.349007 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:15.349674 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:15.349681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:15.350153 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:15.350712 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:15.350730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.351753 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.351757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:15.352929 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:15.352939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.353856 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.353861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:15.354055 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:15.354644 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:15.354668 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:15.356950 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:15.360130 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:15.360138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:15.360152 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:15.360616 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:15.360933 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:15.360937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:15.360940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:15.361135 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:15.361770 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:15.361777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:15.362126 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 05:24:15.362375 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 05:24:15.362806 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":246,"threads_started":1}
I20260928 05:24:15.365629 19199 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43889
I20260928 05:24:15.365931 19199 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:15.366716 19199 heartbeater.cc:511] Master 127.8.124.254:43889 requested a full tablet report, sending...
I20260928 05:24:15.368682 19010 ts_manager.cc:194] Registered new tserver with Master: eedbd051c7bc4847a5c6af6cc16488cb (127.8.124.193:42127)
I20260928 05:24:15.370091 19010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:49101
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:24:15.402330 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:24:15.711402 19205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:15.711651 19205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:15.711771 19205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:15.718936 19205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:15.719027 19205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:15.719142 19205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:24:15.728353 19205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:15.729512 19205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:15.730872 19205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:15.742738 19210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:15.742676 19211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:15.742710 19213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:15.743745 19205 server_base.cc:1034] running on GCE node
I20260928 05:24:15.744273 19205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:15.745100 19205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:15.746282 19205 hybrid_clock.cc:648] HybridClock initialized: now 1790573055746230 us; error 39 us; skew 500 ppm
I20260928 05:24:15.746555 19205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:24:15 dist-test-slave-515x krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:24:15.749660 19205 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:24:15.751231 19205 webserver.cc:460] Webserver started at http://127.8.124.194:42453/ using document root <none> and password file <none>
I20260928 05:24:15.752000 19205 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:15.752130 19205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:15.752421 19205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:15.755942 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "b866986b44a640f48abcd9cc8b7b2a26"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "621549f0eaf24753f62c515b7303efff"
server_key_iv: "fb2909b7baaace862f0dfa55cdab4fbc"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.756845 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "b866986b44a640f48abcd9cc8b7b2a26"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "621549f0eaf24753f62c515b7303efff"
server_key_iv: "fb2909b7baaace862f0dfa55cdab4fbc"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.762708 19205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 05:24:15.766666 19220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.768185 19205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:24:15.768308 19205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "b866986b44a640f48abcd9cc8b7b2a26"
format_stamp: "Formatted at 2026-09-28 05:24:15 on dist-test-slave-515x"
server_key: "621549f0eaf24753f62c515b7303efff"
server_key_iv: "fb2909b7baaace862f0dfa55cdab4fbc"
server_key_version: "encryptionkey@0"
I20260928 05:24:15.768483 19205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:15.782980 19205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:15.786576 19205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:15.786809 19205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:15.787654 19205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:15.788961 19205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:15.789028 19205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.789132 19205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:15.789180 19205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:15.818254 19205 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:43609
I20260928 05:24:15.818297 19332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:43609 every 8 connection(s)
I20260928 05:24:15.819469 19205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:24:15.822151 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 19205
I20260928 05:24:15.822309 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:24:15.822690 8691 external_mini_cluster.cc:1428] Setting key 483f63dac0d86d79dc067b715929c5d5
Sep 28 05:24:15 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:24:15.837976 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:15.821781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:40307 (local address 127.8.124.254:43889)
0928 05:24:15.821969 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:15.821975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:15.823245 (+ 1270us) server_negotiation.cc:407] Connection header received
0928 05:24:15.825170 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:15.825175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:15.825251 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:15.825335 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:15.826800 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.827521 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.828988 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.829376 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.829481 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:15.832821 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:15.832826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:15.832829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:15.832860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:15.835211 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:15.835859 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:15.835862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:15.835865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:15.835981 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:15.836361 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:15.836365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:15.836367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:15.836614 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:15.836735 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:15.837400 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 05:24:15.837651 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:24:15.839005 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:15.822332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43889 (local address 127.8.124.194:40307)
0928 05:24:15.823078 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:15.823131 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:15.824929 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:15.825572 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:15.825580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:15.826098 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:15.826646 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:15.826663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.827655 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.827659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:15.828860 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:15.828871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:15.829571 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:15.829578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:15.829812 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:15.830428 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:15.830449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:15.832616 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:15.835340 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:15.835348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:15.835365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:15.835747 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:15.836078 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:15.836083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:15.836087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:15.836259 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:15.836726 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:15.836734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:15.837063 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 05:24:15.837283 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 05:24:15.837824 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":281,"threads_started":1}
I20260928 05:24:15.840291 19333 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43889
I20260928 05:24:15.840579 19333 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:15.841255 19333 heartbeater.cc:511] Master 127.8.124.254:43889 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:24:15.842660 19010 ts_manager.cc:194] Registered new tserver with Master: b866986b44a640f48abcd9cc8b7b2a26 (127.8.124.194:43609)
I20260928 05:24:15.843484 19010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:40307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:24:15.868808 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:24:16.179591 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:16.179877 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:16.180013 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:16.187211 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:16.187347 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:16.187640 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:24:16.197584 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:16.198872 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:16.200143 19338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:16.212446 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:16.212416 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:16.212479 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:16.213166 19338 server_base.cc:1034] running on GCE node
I20260928 05:24:16.213784 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:16.214692 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:16.215878 19338 hybrid_clock.cc:648] HybridClock initialized: now 1790573056215830 us; error 40 us; skew 500 ppm
I20260928 05:24:16.216168 19338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:24:16 dist-test-slave-515x krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:24:16.219080 19338 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:24:16.220362 19338 webserver.cc:460] Webserver started at http://127.8.124.195:33583/ using document root <none> and password file <none>
I20260928 05:24:16.220990 19338 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:16.221107 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:16.221369 19338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:24:16.224148 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "d0d706e772d44bfd997d6d286a272dd6"
format_stamp: "Formatted at 2026-09-28 05:24:16 on dist-test-slave-515x"
server_key: "99f7ada13594c176e5def22b0f69eb8f"
server_key_iv: "197d8425ab1f34a8ff855c80ca8df697"
server_key_version: "encryptionkey@0"
I20260928 05:24:16.224845 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "d0d706e772d44bfd997d6d286a272dd6"
format_stamp: "Formatted at 2026-09-28 05:24:16 on dist-test-slave-515x"
server_key: "99f7ada13594c176e5def22b0f69eb8f"
server_key_iv: "197d8425ab1f34a8ff855c80ca8df697"
server_key_version: "encryptionkey@0"
I20260928 05:24:16.229652 19338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 05:24:16.232905 19353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:16.234207 19338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:24:16.234346 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "d0d706e772d44bfd997d6d286a272dd6"
format_stamp: "Formatted at 2026-09-28 05:24:16 on dist-test-slave-515x"
server_key: "99f7ada13594c176e5def22b0f69eb8f"
server_key_iv: "197d8425ab1f34a8ff855c80ca8df697"
server_key_version: "encryptionkey@0"
I20260928 05:24:16.234517 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:24:16.255678 19338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:24:16.259701 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:24:16.259944 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:24:16.260810 19338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:24:16.262126 19338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:24:16.262204 19338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:16.262327 19338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:24:16.262379 19338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:16.290598 19338 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:33495
I20260928 05:24:16.290642 19465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:33495 every 8 connection(s)
I20260928 05:24:16.291792 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:24:16.299252 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 19338
I20260928 05:24:16.299429 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:24:16.299929 8691 external_mini_cluster.cc:1428] Setting key b3dd878b1fbeeb5ccff4d8012543c1a5
Sep 28 05:24:16 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:24:16.310141 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.294126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:33715 (local address 127.8.124.254:43889)
0928 05:24:16.294285 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.294289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.295482 (+ 1193us) server_negotiation.cc:407] Connection header received
0928 05:24:16.297230 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.297234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.297302 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.297382 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:16.298753 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.299525 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.301514 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.302118 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.302239 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.305313 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:16.305317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:16.305321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:16.305351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:16.307698 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.308329 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.308332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.308334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.308433 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.308743 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.308746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.308747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.308979 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:16.309083 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.309750 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.309905 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:24:16.311102 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.294609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43889 (local address 127.8.124.195:33715)
0928 05:24:16.295312 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.295357 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.297007 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.297562 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.297569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.298095 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:24:16.298609 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.298624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.299681 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.299686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.301383 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.301398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.302329 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.302334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.302567 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.303160 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:24:16.303185 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:24:16.305138 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:24:16.307833 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:16.307839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:16.307852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:16.308248 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:16.308505 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:24:16.308508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:24:16.308510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:24:16.308672 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:24:16.309073 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:24:16.309078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:24:16.309409 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.309684 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.310063 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":258,"threads_started":1}
I20260928 05:24:16.312296 19466 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43889
I20260928 05:24:16.312626 19466 heartbeater.cc:464] Registering TS with master...
I20260928 05:24:16.313370 19466 heartbeater.cc:511] Master 127.8.124.254:43889 requested a full tablet report, sending...
I20260928 05:24:16.314775 19009 ts_manager.cc:194] Registered new tserver with Master: d0d706e772d44bfd997d6d286a272dd6 (127.8.124.195:33495)
I20260928 05:24:16.315560 19009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:33715
I20260928 05:24:16.326889 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:24:16.342527 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.331197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.8.124.254:43889)
0928 05:24:16.331391 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.331396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.331721 (+ 325us) server_negotiation.cc:407] Connection header received
0928 05:24:16.331870 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.331875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.331934 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.332007 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:16.333304 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.334157 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.335297 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.335733 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.335831 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.337896 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:16.337903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:16.337906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:16.337936 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:16.340279 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.340824 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.340828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.340830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.340949 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.341238 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.341242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.341244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.341562 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:16.341770 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.342015 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.342165 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:24:16.346441 19009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:24:16.352847 19009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:24:16.373250 19199 heartbeater.cc:503] Master 127.8.124.254:43889 was elected leader, sending a full tablet report...
I20260928 05:24:16.385381 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.363785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.8.124.194:43609)
0928 05:24:16.377117 (+ 13332us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.377127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.377159 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:24:16.377269 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.377278 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.377548 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.377765 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:24:16.378471 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.380547 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.383064 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.384332 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.384450 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.384657 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.384939 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.385098 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13156,"thread_start_us":14181,"threads_started":1}
I20260928 05:24:16.385427 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.363700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43609 (local address 127.0.0.1:56202)
0928 05:24:16.376297 (+ 12597us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.376342 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.376539 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.377944 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.377951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.377982 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:24:16.378303 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.378313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.380674 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.380679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.382887 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.382901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.384539 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.384546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.384736 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.384773 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.384898 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.385096 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12292,"thread_start_us":12135,"threads_started":1}
I20260928 05:24:16.387354 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.364420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.8.124.195:33495)
0928 05:24:16.377944 (+ 13524us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.377952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.378814 (+ 862us) server_negotiation.cc:407] Connection header received
0928 05:24:16.378913 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.378920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.379133 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.379290 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:24:16.381452 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.382749 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.385071 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.386400 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.386504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.386712 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.386875 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.387040 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13378,"thread_start_us":13220,"threads_started":1}
I20260928 05:24:16.387765 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.363831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:33495 (local address 127.0.0.1:52438)
0928 05:24:16.378556 (+ 14725us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.378572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.378679 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.380793 (+ 2114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.380798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.380824 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:24:16.381266 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.381281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.382876 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.382881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.384887 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.384901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.386566 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.386572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.386721 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.386752 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.386830 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.386977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14596,"thread_start_us":14456,"threads_started":1}
I20260928 05:24:16.389398 19268 tablet_service.cc:1467] Processing CreateTablet for tablet da02a35fba8a44c9abf853bd6389b57b (DEFAULT_TABLE table=test-table [id=928978b1784b4d7298e55e9a73c4e467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:24:16.390723 19268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02a35fba8a44c9abf853bd6389b57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:16.391577 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.378718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:42127 (local address 127.0.0.1:36852)
0928 05:24:16.379871 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.379886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.379975 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.381868 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.381873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.381897 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:24:16.382257 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.382267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.383996 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.384002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.386059 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.386073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.390825 (+ 4752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.390832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.390989 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.391020 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.391126 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.391287 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":917,"threads_started":1}
I20260928 05:24:16.391670 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.379003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.8.124.193:42127)
0928 05:24:16.381320 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.381327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.381350 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:24:16.381428 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.381435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.381668 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.381862 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:24:16.382431 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.383863 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.386197 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.390623 (+ 4426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.390760 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.390973 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.391171 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.391373 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2167,"thread_start_us":2015,"threads_started":1}
I20260928 05:24:16.391330 19401 tablet_service.cc:1467] Processing CreateTablet for tablet da02a35fba8a44c9abf853bd6389b57b (DEFAULT_TABLE table=test-table [id=928978b1784b4d7298e55e9a73c4e467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:24:16.392736 19401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02a35fba8a44c9abf853bd6389b57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:16.395320 19134 tablet_service.cc:1467] Processing CreateTablet for tablet da02a35fba8a44c9abf853bd6389b57b (DEFAULT_TABLE table=test-table [id=928978b1784b4d7298e55e9a73c4e467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:24:16.396701 19134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02a35fba8a44c9abf853bd6389b57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:24:16.403604 19484 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Bootstrap starting.
I20260928 05:24:16.405526 19485 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Bootstrap starting.
I20260928 05:24:16.408262 19484 tablet_bootstrap.cc:654] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:16.408841 19485 tablet_bootstrap.cc:654] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:16.409477 19484 log.cc:824] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:16.409952 19486 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Bootstrap starting.
I20260928 05:24:16.409991 19485 log.cc:824] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:16.412870 19486 tablet_bootstrap.cc:654] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Neither blocks nor log segments found. Creating new log.
I20260928 05:24:16.413200 19485 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: No bootstrap required, opened a new log
I20260928 05:24:16.413487 19485 ts_tablet_manager.cc:1397] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:24:16.413851 19486 log.cc:824] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Log is configured to *not* fsync() on all Append() calls
I20260928 05:24:16.419572 19486 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: No bootstrap required, opened a new log
I20260928 05:24:16.419574 19484 tablet_bootstrap.cc:492] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: No bootstrap required, opened a new log
I20260928 05:24:16.419840 19486 ts_tablet_manager.cc:1397] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 05:24:16.419842 19484 ts_tablet_manager.cc:1397] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260928 05:24:16.421808 19485 raft_consensus.cc:348] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.422148 19485 raft_consensus.cc:374] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:16.422250 19485 raft_consensus.cc:729] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b866986b44a640f48abcd9cc8b7b2a26, State: Initialized, Role: FOLLOWER
I20260928 05:24:16.422791 19485 consensus_queue.cc:260] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.423763 19333 heartbeater.cc:503] Master 127.8.124.254:43889 was elected leader, sending a full tablet report...
I20260928 05:24:16.424468 19485 ts_tablet_manager.cc:1428] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 05:24:16.429929 19484 raft_consensus.cc:348] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.429929 19486 raft_consensus.cc:348] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.430349 19486 raft_consensus.cc:374] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:16.430385 19484 raft_consensus.cc:374] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:24:16.430455 19486 raft_consensus.cc:729] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedbd051c7bc4847a5c6af6cc16488cb, State: Initialized, Role: FOLLOWER
I20260928 05:24:16.430464 19484 raft_consensus.cc:729] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d706e772d44bfd997d6d286a272dd6, State: Initialized, Role: FOLLOWER
I20260928 05:24:16.431052 19486 consensus_queue.cc:260] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.431043 19484 consensus_queue.cc:260] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.432027 19466 heartbeater.cc:503] Master 127.8.124.254:43889 was elected leader, sending a full tablet report...
I20260928 05:24:16.432717 19484 ts_tablet_manager.cc:1428] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 05:24:16.432988 19486 ts_tablet_manager.cc:1428] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 05:24:16.543241 19467 tablet.cc:2369] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:24:16.571198 19334 tablet.cc:2369] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:24:16.595973 19200 tablet.cc:2369] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:24:16.676788 19492 raft_consensus.cc:482] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:24:16.677073 19492 raft_consensus.cc:504] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.678716 19492 leader_election.cc:288] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b866986b44a640f48abcd9cc8b7b2a26 (127.8.124.194:43609), d0d706e772d44bfd997d6d286a272dd6 (127.8.124.195:33495)
I20260928 05:24:16.685072 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.679132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:39099 (local address 127.8.124.195:33495)
0928 05:24:16.679327 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.679333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.679980 (+ 647us) server_negotiation.cc:407] Connection header received
0928 05:24:16.680152 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.680155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.680212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.680282 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:24:16.680905 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.681648 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.683476 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.684244 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.684377 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.684448 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.684712 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.684828 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:24:16.685168 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.679091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:33495 (local address 127.8.124.193:39099)
0928 05:24:16.679868 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.679885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.680012 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.680304 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.680308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.680335 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:24:16.680770 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.680779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.681756 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.681761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.683339 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.683350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.684422 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.684428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.684573 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.684595 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.684691 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.684809 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":93,"thread_start_us":435,"threads_started":1}
I20260928 05:24:16.685181 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.679219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:54023 (local address 127.8.124.194:43609)
0928 05:24:16.679455 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.679461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.680040 (+ 579us) server_negotiation.cc:407] Connection header received
0928 05:24:16.680209 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.680213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.680266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.680333 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:24:16.680995 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.681679 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.683469 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.684198 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.684459 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.684529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.684821 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.684953 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:24:16.685839 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.679177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:43609 (local address 127.8.124.193:54023)
0928 05:24:16.679956 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 05:24:16.679973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:24:16.680092 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:24:16.680359 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:24:16.680363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:24:16.680435 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:24:16.680844 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.680854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.681798 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.681806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.683325 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:24:16.683335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.684516 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.684523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:24:16.684645 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.684668 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:24:16.685411 (+ 743us) client_negotiation.cc:240] Negotiation successful
0928 05:24:16.685524 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":39,"thread_start_us":488,"threads_started":1}
I20260928 05:24:16.686127 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02a35fba8a44c9abf853bd6389b57b" candidate_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d706e772d44bfd997d6d286a272dd6" is_pre_election: true
I20260928 05:24:16.686744 19421 raft_consensus.cc:2454] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedbd051c7bc4847a5c6af6cc16488cb in term 0.
I20260928 05:24:16.686738 19288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02a35fba8a44c9abf853bd6389b57b" candidate_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b866986b44a640f48abcd9cc8b7b2a26" is_pre_election: true
I20260928 05:24:16.687162 19288 raft_consensus.cc:2454] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedbd051c7bc4847a5c6af6cc16488cb in term 0.
I20260928 05:24:16.687408 19089 leader_election.cc:302] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0d706e772d44bfd997d6d286a272dd6, eedbd051c7bc4847a5c6af6cc16488cb; no voters:
I20260928 05:24:16.687837 19492 raft_consensus.cc:2781] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:24:16.687976 19492 raft_consensus.cc:482] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:24:16.688058 19492 raft_consensus.cc:3037] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:16.690241 19492 raft_consensus.cc:504] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.690863 19492 leader_election.cc:288] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [CANDIDATE]: Term 1 election: Requested vote from peers b866986b44a640f48abcd9cc8b7b2a26 (127.8.124.194:43609), d0d706e772d44bfd997d6d286a272dd6 (127.8.124.195:33495)
I20260928 05:24:16.691231 19288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02a35fba8a44c9abf853bd6389b57b" candidate_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b866986b44a640f48abcd9cc8b7b2a26"
I20260928 05:24:16.691347 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02a35fba8a44c9abf853bd6389b57b" candidate_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d706e772d44bfd997d6d286a272dd6"
I20260928 05:24:16.691396 19288 raft_consensus.cc:3037] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:16.691527 19421 raft_consensus.cc:3037] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:24:16.693490 19288 raft_consensus.cc:2454] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedbd051c7bc4847a5c6af6cc16488cb in term 1.
I20260928 05:24:16.693495 19421 raft_consensus.cc:2454] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedbd051c7bc4847a5c6af6cc16488cb in term 1.
I20260928 05:24:16.693854 19090 leader_election.cc:302] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b866986b44a640f48abcd9cc8b7b2a26, eedbd051c7bc4847a5c6af6cc16488cb; no voters:
I20260928 05:24:16.694133 19492 raft_consensus.cc:2781] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:24:16.694531 19492 raft_consensus.cc:686] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 1 LEADER]: Becoming Leader. State: Replica: eedbd051c7bc4847a5c6af6cc16488cb, State: Running, Role: LEADER
I20260928 05:24:16.695003 19492 consensus_queue.cc:237] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } }
I20260928 05:24:16.699329 19010 catalog_manager.cc:5495] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb reported cstate change: term changed from 0 to 1, leader changed from <none> to eedbd051c7bc4847a5c6af6cc16488cb (127.8.124.193). New cstate: current_term: 1 leader_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eedbd051c7bc4847a5c6af6cc16488cb" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 42127 } health_report { overall_health: HEALTHY } } }
I20260928 05:24:16.749039 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.744271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.8.124.193:42127)
0928 05:24:16.744480 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.744485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.744508 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:24:16.744704 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.744707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.744795 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.744918 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:24:16.745186 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.745933 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.747156 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.747553 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.747645 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.748039 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:24:16.748526 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:24:16.748631 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.748757 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.748819 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:24:16.757292 19421 raft_consensus.cc:1261] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 1 FOLLOWER]: Refusing update from remote peer eedbd051c7bc4847a5c6af6cc16488cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:24:16.757293 19288 raft_consensus.cc:1261] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 1 FOLLOWER]: Refusing update from remote peer eedbd051c7bc4847a5c6af6cc16488cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:24:16.758319 19495 consensus_queue.cc:1035] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b866986b44a640f48abcd9cc8b7b2a26" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:24:16.758631 19492 consensus_queue.cc:1035] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d706e772d44bfd997d6d286a272dd6" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:24:16.786883 19010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:24:16.787117 19010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:24:16.790220 19010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf79a364b09e4f85a36dff5a6080c6f3: Sending DeleteTablet for 3 replicas of tablet da02a35fba8a44c9abf853bd6389b57b
I20260928 05:24:16.791224 19268 tablet_service.cc:1514] Processing DeleteTablet for tablet da02a35fba8a44c9abf853bd6389b57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:24:16 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:56202
I20260928 05:24:16.791565 19401 tablet_service.cc:1514] Processing DeleteTablet for tablet da02a35fba8a44c9abf853bd6389b57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:24:16 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:52438
I20260928 05:24:16.791726 19134 tablet_service.cc:1514] Processing DeleteTablet for tablet da02a35fba8a44c9abf853bd6389b57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:24:16 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:36852
I20260928 05:24:16.793359 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 19071
I20260928 05:24:16.802918 19509 tablet_replica.cc:330] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: stopping tablet replica
I20260928 05:24:16.803485 19508 tablet_replica.cc:330] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: stopping tablet replica
I20260928 05:24:16.803462 19509 raft_consensus.cc:2229] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:24:16.803984 19509 raft_consensus.cc:2258] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:24:16.804041 19508 raft_consensus.cc:2229] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:24:16.804550 19508 raft_consensus.cc:2258] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:24:16.804708 19510 tablet_replica.cc:330] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: stopping tablet replica
I20260928 05:24:16.805109 19510 raft_consensus.cc:2229] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:24:16.805809 19510 raft_consensus.cc:2258] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:24:16.806458 19508 ts_tablet_manager.cc:1905] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:24:16.807706 19510 ts_tablet_manager.cc:1905] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:24:16.809012 19509 ts_tablet_manager.cc:1905] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:24:16.813521 19508 ts_tablet_manager.cc:1918] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:24:16.813686 19508 log.cc:1196] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/da02a35fba8a44c9abf853bd6389b57b
I20260928 05:24:16.814198 19508 ts_tablet_manager.cc:1939] T da02a35fba8a44c9abf853bd6389b57b P b866986b44a640f48abcd9cc8b7b2a26: Deleting consensus metadata
I20260928 05:24:16.815126 19510 ts_tablet_manager.cc:1918] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:24:16.815318 19510 log.cc:1196] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/da02a35fba8a44c9abf853bd6389b57b
I20260928 05:24:16.815815 19510 ts_tablet_manager.cc:1939] T da02a35fba8a44c9abf853bd6389b57b P eedbd051c7bc4847a5c6af6cc16488cb: Deleting consensus metadata
I20260928 05:24:16.816663 18997 catalog_manager.cc:4841] TS b866986b44a640f48abcd9cc8b7b2a26 (127.8.124.194:43609): tablet da02a35fba8a44c9abf853bd6389b57b (table test-table [id=928978b1784b4d7298e55e9a73c4e467]) successfully deleted
I20260928 05:24:16.817258 18996 catalog_manager.cc:4841] TS eedbd051c7bc4847a5c6af6cc16488cb (127.8.124.193:42127): tablet da02a35fba8a44c9abf853bd6389b57b (table test-table [id=928978b1784b4d7298e55e9a73c4e467]) successfully deleted
I20260928 05:24:16.821070 19509 ts_tablet_manager.cc:1918] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:24:16.821244 19509 log.cc:1196] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/da02a35fba8a44c9abf853bd6389b57b
I20260928 05:24:16.821769 19509 ts_tablet_manager.cc:1939] T da02a35fba8a44c9abf853bd6389b57b P d0d706e772d44bfd997d6d286a272dd6: Deleting consensus metadata
I20260928 05:24:16.823701 18996 catalog_manager.cc:4841] TS d0d706e772d44bfd997d6d286a272dd6 (127.8.124.195:33495): tablet da02a35fba8a44c9abf853bd6389b57b (table test-table [id=928978b1784b4d7298e55e9a73c4e467]) successfully deleted
Sep 28 05:24:16 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:24:16.835615 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.796302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.8.124.193:42127)
0928 05:24:16.796458 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.796464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.817676 (+ 21212us) server_negotiation.cc:407] Connection header received
0928 05:24:16.817954 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.817959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.818064 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.818128 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:16.819737 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.820926 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.822686 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.823399 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.823491 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.828675 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:16.828689 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:16.828707 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:16.828757 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:16.832419 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.833210 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.833219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.833222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.833389 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.833814 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.833819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.833823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.834206 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:16.834479 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.834839 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.835075 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:24:16.836306 19194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:16.934288 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 19071
I20260928 05:24:16.949110 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 19205
Sep 28 05:24:16 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:24:16.963258 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:16.949805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.8.124.194:43609)
0928 05:24:16.949984 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:16.949989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:16.950007 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:24:16.950119 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:16.950123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:16.950192 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:16.950272 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:16.951437 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.952169 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.953418 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:16.953907 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:16.954007 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:16.957946 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:16.957960 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:16.957978 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:16.958013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:16.960765 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.961308 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.961358 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.961361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.961487 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:16.961902 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:16.961906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:16.961908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:16.962280 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:16.962545 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:16.962765 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 05:24:16.962929 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:24:16.963898 19328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:17.050143 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 19205
I20260928 05:24:17.063247 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 19338
Sep 28 05:24:17 dist-test-slave-515x krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:24:17.077153 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:24:17.063978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.8.124.195:33495)
0928 05:24:17.064153 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:24:17.064159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:24:17.064178 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:24:17.064345 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:24:17.064350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:24:17.064416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:24:17.064507 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:24:17.065814 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:17.066521 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:17.067772 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:24:17.068165 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:24:17.068270 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:24:17.072237 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:24:17.072251 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:24:17.072267 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:24:17.072304 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:24:17.074709 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:17.075290 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:17.075298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:17.075300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:17.075432 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:24:17.075747 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:24:17.075751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:24:17.075753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:24:17.076098 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:24:17.076349 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 05:24:17.076658 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 05:24:17.076852 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:24:17.077754 19461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:17.163601 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 19338
I20260928 05:24:17.176652 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 18978
I20260928 05:24:17.177727 19040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:24:17.282811 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 18978
2026-09-28T05:24:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4142 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:24:17 dist-test-slave-515x krb5kdc[19518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:24:17 dist-test-slave-515x krb5kdc[19518](info): set up 2 sockets
Sep 28 05:24:17 dist-test-slave-515x krb5kdc[19518](info): commencing operation
krb5kdc: starting...
W20260928 05:24:18.294203 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:24:18 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:24:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangeradmin_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangeradmin_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangerlookup_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangerlookup_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
I20260928 05:24:18.582697 8691 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/postgres -l logfile start
2026-09-28 05:24:20.894 UTC [19548] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:24:20.894 UTC [19548] LOG: listening on IPv4 address "127.8.124.212", port 34373
2026-09-28 05:24:20.897 UTC [19548] LOG: listening on Unix socket "/tmp/.s.PGSQL.34373"
2026-09-28 05:24:20.917 UTC [19551] LOG: database system was shut down at 2026-09-28 05:24:19 UTC
2026-09-28 05:24:20.920 UTC [19548] LOG: database system is ready to accept connections
I20260928 05:24:21.817888 8691 mini_postgres.cc:96] Postgres bound to 34373
2026-09-28 05:24:21.833 UTC [19559] FATAL: database "slave" does not exist
127.8.124.212:34373 - accepting connections
I20260928 05:24:21.834182 8691 mini_ranger.cc:161] Starting Ranger...
I20260928 05:24:21.856202 8691 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:24:22.291368 8691 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:24:22.291483 8691 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin
2026-09-28 05:24:22,563 [I] DB FLAVOR :POSTGRES
2026-09-28 05:24:22,563 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:24:22,564 [I] Checking connection..
2026-09-28 05:24:22,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:24:22,802 [I] Checking connection passed.
2026-09-28 05:24:22,803 [I] --------- Verifying version history table ---------
2026-09-28 05:24:22,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:24:23,052 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:24:23,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:24:23,297 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:24:23,298 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:24:23,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:24:23.537 UTC [19623] WARNING: there is no transaction in progress
2026-09-28 05:24:23,543 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:24:23,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:24:23,785 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:24:23,785 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:24:23,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:24:24,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:24:24,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-515x.c.gcp-upstream.internal','N') ;"
2026-09-28 05:24:24,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:24:24,753 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:24:24,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:24:24,999 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:24:25,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:24:25,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:24:25,481 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:24:25,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:24:26.372 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.388 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.403 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.418 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.433 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.452 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.504 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.510 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.523 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.533 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.543 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.555 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.564 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.573 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.584 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:26.590 UTC [19731] WARNING: there is no transaction in progress
2026-09-28 05:24:27,278 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:24:27,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:24:27,492 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:24:27,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:24:27,731 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:24:27,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:24:27,982 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:24:27,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:24:28,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:24:28,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-515x.c.gcp-upstream.internal';"
2026-09-28 05:24:28,718 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:24:28,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:24:28,964 [I] DB_PATCHES have already been applied
I20260928 05:24:28.970470 8691 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbMG2Xa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:24:28.970551 8691 mini_ranger.cc:193] Using host: 127.8.124.212
I20260928 05:24:28.983724 8691 mini_ranger.cc:227] Ranger admin URL: http://127.8.124.212:55103
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:24:30.008734 8691 mini_ranger.cc:160] Time spent starting Ranger: real 8.175s user 0.001s sys 0.034s
Sep 28, 2026 5:24:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:24:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:24:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:24:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:24:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:24:51.093050 8691 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangerkms_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/rangerkms_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.124.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.124.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:24:51 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:24:51.283701 8691 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:24:51.302560 8691 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:24:52.077915 8691 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:24:52.078063 8691 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms
I20260928 05:24:52.078159 8691 mini_ranger_kms.cc:227] Using postgres at 127.8.124.212:34373
2026-09-28 05:24:52,454 [I] DB FLAVOR :POSTGRES
2026-09-28 05:24:52,454 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:24:52,455 [I] Checking connection
2026-09-28 05:24:52,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:24:52,711 [I] connection success
2026-09-28 05:24:52,711 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:24:52,711 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:24:52,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:24:52,932 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:24:52,933 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:24:52,933 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:24:52,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:34373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:24:53,174 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:24:53.632369 8691 mini_ranger_kms.cc:310] Created kms service
I20260928 05:24:53.814962 8691 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:24:53.855707 8691 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:24:53 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:24:54.262305 8691 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:24:54.262598 8691 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:24:54.262650 8691 mini_ranger_kms.cc:240] Using host: 127.8.124.212
I20260928 05:24:54.282644 8691 mini_ranger_kms.cc:276] Ranger KMS PID: 19911
I20260928 05:24:54.282794 8691 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.124.212:59819
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:24:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 05:24:56.313459 8691 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.030s user 0.103s sys 0.287s
I20260928 05:24:56.313738 8691 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:24:56.313946 8691 mini_ranger_kms.cc:401] 127.8.124.212:59819/kms/v1/keys
Sep 28 05:24:56 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790573096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 05:24:56 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Sep 28 05:24:57 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573096, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Sep 28 05:24:59 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:24:59.521513 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33941
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33941
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59819/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:24:59.836618 19962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:24:59.836882 19962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:24:59.836987 19962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:24:59.844120 19962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:24:59.844224 19962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:24:59.844286 19962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:24:59.844365 19962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:24:59.844424 19962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:24:59.854463 19962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.124.212:59819/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33941
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:24:59.855685 19962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:24:59.856962 19962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:24:59.870009 19967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:59.870029 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:24:59.870625 19970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:24:59.874966 19962 server_base.cc:1034] running on GCE node
I20260928 05:24:59.875707 19962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:24:59.877115 19962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:24:59.878321 19962 hybrid_clock.cc:648] HybridClock initialized: now 1790573099878280 us; error 38 us; skew 500 ppm
I20260928 05:24:59.878635 19962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:24:59 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:24:59.886952 19962 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:24:59.888622 19962 webserver.cc:460] Webserver started at http://127.8.124.254:34855/ using document root <none> and password file <none>
I20260928 05:24:59.889300 19962 fs_manager.cc:359] Metadata directory not provided
I20260928 05:24:59.889403 19962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:24:59.889729 19962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:24:59 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:24:59.951174 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "ec05c99dd733431c8f33e95fe3766c3a"
format_stamp: "Formatted at 2026-09-28 05:24:59 on dist-test-slave-515x"
server_key: "09638d52bbbb083a3c47a7b2cf3c59b7"
server_key_iv: "e8fbe2d340be8501d843517a9e8d5053"
server_key_version: "kuduclusterkey@0"
I20260928 05:24:59.952001 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "ec05c99dd733431c8f33e95fe3766c3a"
format_stamp: "Formatted at 2026-09-28 05:24:59 on dist-test-slave-515x"
server_key: "09638d52bbbb083a3c47a7b2cf3c59b7"
server_key_iv: "e8fbe2d340be8501d843517a9e8d5053"
server_key_version: "kuduclusterkey@0"
I20260928 05:24:59.957436 19962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:24:59.972651 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:24:59.974148 19962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:24:59.974319 19962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "ec05c99dd733431c8f33e95fe3766c3a"
format_stamp: "Formatted at 2026-09-28 05:24:59 on dist-test-slave-515x"
server_key: "09638d52bbbb083a3c47a7b2cf3c59b7"
server_key_iv: "e8fbe2d340be8501d843517a9e8d5053"
server_key_version: "kuduclusterkey@0"
I20260928 05:24:59.974525 19962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:00.013581 19962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:00.017571 19962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:00.017884 19962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:00.034027 19962 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:33941
I20260928 05:25:00.034050 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:33941 every 8 connection(s)
I20260928 05:25:00.035287 19962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:25:00.035665 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 19962
I20260928 05:25:00.035853 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:25:00.036602 8691 external_mini_cluster.cc:1428] Setting key 2349a77891912210166d8d98e516739d
I20260928 05:25:00.039438 20029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:25:00 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:00.052039 20029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: Bootstrap starting.
I20260928 05:25:00.055222 20029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:00.056146 20029 log.cc:824] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:00.058846 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:00.038920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.8.124.254:33941)
0928 05:25:00.039741 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:00.039756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:00.040310 (+ 554us) server_negotiation.cc:407] Connection header received
0928 05:25:00.042072 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:00.042093 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:00.042469 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:00.042790 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:00.044264 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:00.045666 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:00.047367 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:00.047992 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:00.048106 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:00.052304 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:00.052312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:00.052326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:00.052365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:00.055105 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:00.055766 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:00.055773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:00.055778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:00.055942 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:00.056285 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:00.056290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:00.056293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:00.056786 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:00.057083 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:00.057505 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 05:25:00.057899 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":368,"threads_started":1}
I20260928 05:25:00.058866 20029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: No bootstrap required, opened a new log
I20260928 05:25:00.066274 20029 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } }
I20260928 05:25:00.066619 20029 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:00.066712 20029 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec05c99dd733431c8f33e95fe3766c3a, State: Initialized, Role: FOLLOWER
I20260928 05:25:00.067196 20029 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } }
I20260928 05:25:00.067400 20029 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:25:00.067513 20029 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:25:00.067648 20029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:00.069590 20029 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } }
I20260928 05:25:00.070055 20029 leader_election.cc:302] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec05c99dd733431c8f33e95fe3766c3a; no voters:
I20260928 05:25:00.070581 20029 leader_election.cc:288] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:25:00.070696 20034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:25:00.071300 20034 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [term 1 LEADER]: Becoming Leader. State: Replica: ec05c99dd733431c8f33e95fe3766c3a, State: Running, Role: LEADER
I20260928 05:25:00.071837 20034 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } }
I20260928 05:25:00.072479 20029 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:25:00.074491 20036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec05c99dd733431c8f33e95fe3766c3a. Latest consensus state: current_term: 1 leader_uuid: "ec05c99dd733431c8f33e95fe3766c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } } }
I20260928 05:25:00.074635 20035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec05c99dd733431c8f33e95fe3766c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec05c99dd733431c8f33e95fe3766c3a" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33941 } } }
I20260928 05:25:00.074856 20036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:00.074910 20035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:00.076156 20042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:25:00.080647 20042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:25:00.088976 20029 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:25:00.090309 20042 catalog_manager.cc:1347] Generated new cluster ID: b60d360cd67549b1bb28fa2c6f7eeb79
I20260928 05:25:00.090405 20042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:25:00.133184 20042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:25:00.135298 20042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:25:00.155145 20042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: Generated new TSK 0
I20260928 05:25:00.156334 20042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:25:00.861819 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:25:01.282750 20074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:01.283140 20074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:01.283305 20074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:01.295349 20074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:01.295545 20074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:01.295768 20074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:25:01.312050 20074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.124.212:59819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:01.313830 20074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:01.315970 20074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:01.342579 20085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:01.344584 20084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:01.353829 20074 server_base.cc:1034] running on GCE node
W20260928 05:25:01.354743 20087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:01.355682 20074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:01.357041 20074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:01.358336 20074 hybrid_clock.cc:648] HybridClock initialized: now 1790573101358269 us; error 53 us; skew 500 ppm
I20260928 05:25:01.358735 20074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:01.363145 20074 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:25:01.366484 20074 webserver.cc:460] Webserver started at http://127.8.124.193:46837/ using document root <none> and password file <none>
I20260928 05:25:01.367363 20074 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:01.367532 20074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:01.367935 20074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:01.385617 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "d9419698b0ad4a508bcdb0a6442c8dbc"
format_stamp: "Formatted at 2026-09-28 05:25:01 on dist-test-slave-515x"
server_key: "6188d125a53a2a93c805e2a47b49dff3"
server_key_iv: "75c355ca68980d6c890f6481f14dfe41"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:01.386714 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "d9419698b0ad4a508bcdb0a6442c8dbc"
format_stamp: "Formatted at 2026-09-28 05:25:01 on dist-test-slave-515x"
server_key: "6188d125a53a2a93c805e2a47b49dff3"
server_key_iv: "75c355ca68980d6c890f6481f14dfe41"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:01.394418 20074 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 05:25:01.426179 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:01.428303 20074 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260928 05:25:01.428503 20074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "d9419698b0ad4a508bcdb0a6442c8dbc"
format_stamp: "Formatted at 2026-09-28 05:25:01 on dist-test-slave-515x"
server_key: "6188d125a53a2a93c805e2a47b49dff3"
server_key_iv: "75c355ca68980d6c890f6481f14dfe41"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:01.428740 20074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:01.459110 20074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:01.464623 20074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:01.465027 20074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:01.466243 20074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:01.467640 20074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:01.467715 20074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:01.467836 20074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:01.467891 20074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:01.530598 20074 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33687
I20260928 05:25:01.530634 20206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33687 every 8 connection(s)
I20260928 05:25:01.532295 20074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:25:01.533396 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 20074
I20260928 05:25:01.533636 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:25:01.534514 8691 external_mini_cluster.cc:1428] Setting key 4ba2fb0f8f1000b9e22fc88e5163f5d9
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:25:01.570915 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:01.536977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:35515 (local address 127.8.124.254:33941)
0928 05:25:01.549989 (+ 13012us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:01.549995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:01.550020 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:25:01.550177 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:01.550183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:01.550297 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:01.550375 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:01.552593 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:01.553705 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:01.556481 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:01.557093 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:01.557180 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:01.561312 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:01.561318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:01.561322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:01.561361 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:01.566997 (+ 5636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:01.568096 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:01.568102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:01.568106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:01.568267 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:01.568812 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:01.568817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:01.568820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:01.569149 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:01.569296 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:01.570327 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0928 05:25:01.570548 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12888,"thread_start_us":12778,"threads_started":1}
I20260928 05:25:01.573485 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:01.536398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33941 (local address 127.8.124.193:35515)
0928 05:25:01.546780 (+ 10382us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:01.546843 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:01.549399 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:01.550983 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:01.550997 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:01.551572 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:01.552393 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:01.552418 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:01.553918 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:01.553923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:01.556319 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:01.556334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:01.557505 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:01.557512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:01.557830 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:01.558561 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:01.558594 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:01.561053 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:01.567174 (+ 6121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:01.567185 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:01.567204 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:01.567938 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:01.568404 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:01.568409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:01.568412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:01.568664 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:01.569370 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:01.569379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:01.569876 (+ 497us) client_negotiation.cc:769] Sending connection context
0928 05:25:01.571540 (+ 1664us) client_negotiation.cc:240] Negotiation successful
0928 05:25:01.572089 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9782,"thread_start_us":9540,"threads_started":1}
I20260928 05:25:01.575577 20207 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33941
I20260928 05:25:01.575899 20207 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:01.576653 20207 heartbeater.cc:511] Master 127.8.124.254:33941 requested a full tablet report, sending...
I20260928 05:25:01.579037 19994 ts_manager.cc:194] Registered new tserver with Master: d9419698b0ad4a508bcdb0a6442c8dbc (127.8.124.193:33687)
I20260928 05:25:01.580927 19994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:35515
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:25:01.605396 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:01 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573101, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
W20260928 05:25:02.054641 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:02.055111 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:02.055307 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:02.067097 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:02.067315 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:02.067561 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:25:02.082957 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.124.212:59819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:02.084861 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:02.086658 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:02.103190 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:02.104768 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:02.112299 20213 server_base.cc:1034] running on GCE node
W20260928 05:25:02.112403 20224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:02.113277 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:02.114529 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:02.115799 20213 hybrid_clock.cc:648] HybridClock initialized: now 1790573102115737 us; error 43 us; skew 500 ppm
I20260928 05:25:02.116320 20213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:02.120410 20213 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:25:02.126227 20213 webserver.cc:460] Webserver started at http://127.8.124.194:35753/ using document root <none> and password file <none>
I20260928 05:25:02.127319 20213 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:02.127548 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:02.128171 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:02.151656 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "9c1f9b7146354b6180fd41d70c7ab222"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "90129a574b4e92696976c7ff3ddc0aec"
server_key_iv: "3ee65bf6abae5e76070488303e4bc343"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.152814 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1f9b7146354b6180fd41d70c7ab222"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "90129a574b4e92696976c7ff3ddc0aec"
server_key_iv: "3ee65bf6abae5e76070488303e4bc343"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.158361 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:25:02.177549 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.179819 20213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:25:02.180001 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "9c1f9b7146354b6180fd41d70c7ab222"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "90129a574b4e92696976c7ff3ddc0aec"
server_key_iv: "3ee65bf6abae5e76070488303e4bc343"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.180245 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:02.211091 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:02.216526 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:02.216890 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:02.218084 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:02.219915 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:02.220009 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.220181 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:02.220273 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.263332 20213 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:33877
I20260928 05:25:02.263357 20349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:33877 every 8 connection(s)
I20260928 05:25:02.264976 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:25:02.270989 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 20213
I20260928 05:25:02.271170 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:25:02.271736 8691 external_mini_cluster.cc:1428] Setting key ba38b07d6164b843435cedd517f620c6
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:02.307667 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.267942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:53331 (local address 127.8.124.254:33941)
0928 05:25:02.285153 (+ 17211us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:02.285161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:02.286590 (+ 1429us) server_negotiation.cc:407] Connection header received
0928 05:25:02.288718 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:02.288724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:02.288829 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:02.288929 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:02.291094 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.292179 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.294708 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.295304 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.295395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.300401 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:02.300407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:02.300413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:02.300453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:02.303903 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.304873 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.304879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.304883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.305029 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.305542 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.305547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.305551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.305944 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:02.306069 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:02.307053 (+ 984us) server_negotiation.cc:299] Negotiation successful
0928 05:25:02.307280 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17067,"thread_start_us":18649,"threads_started":1}
I20260928 05:25:02.310249 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.268445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33941 (local address 127.8.124.194:53331)
0928 05:25:02.285804 (+ 17359us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:02.285862 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:02.288288 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:02.289347 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:02.289360 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:02.290006 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:02.290885 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.290912 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.292373 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.292379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.294549 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.294563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.295756 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.295764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.296150 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.296912 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:02.296947 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:02.300149 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:02.304109 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:02.304119 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:02.304139 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:02.304716 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:02.305176 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:02.305181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:02.305185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:02.305418 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:02.306157 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:02.306173 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:02.306608 (+ 435us) client_negotiation.cc:769] Sending connection context
0928 05:25:02.308168 (+ 1560us) client_negotiation.cc:240] Negotiation successful
0928 05:25:02.308763 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17004,"thread_start_us":16797,"threads_started":1}
I20260928 05:25:02.312073 20350 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33941
I20260928 05:25:02.312381 20350 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:02.313199 20350 heartbeater.cc:511] Master 127.8.124.254:33941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
I20260928 05:25:02.314898 19994 ts_manager.cc:194] Registered new tserver with Master: 9c1f9b7146354b6180fd41d70c7ab222 (127.8.124.194:33877)
I20260928 05:25:02.316008 19994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:53331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:25:02.349591 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:25:02.584453 20207 heartbeater.cc:503] Master 127.8.124.254:33941 was elected leader, sending a full tablet report...
W20260928 05:25:02.763183 20356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:02.763458 20356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:02.763568 20356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:02.770888 20356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:02.770987 20356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:02.771116 20356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:25:02.781553 20356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.124.212:59819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:02.782838 20356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:02.784125 20356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:02.796912 20361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:02.796839 20362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:02.797556 20356 server_base.cc:1034] running on GCE node
W20260928 05:25:02.797639 20364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:02.798297 20356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:02.799316 20356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:02.800525 20356 hybrid_clock.cc:648] HybridClock initialized: now 1790573102800468 us; error 51 us; skew 500 ppm
I20260928 05:25:02.800817 20356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:02.804354 20356 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:25:02.805910 20356 webserver.cc:460] Webserver started at http://127.8.124.195:33913/ using document root <none> and password file <none>
I20260928 05:25:02.806551 20356 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:02.806664 20356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:02.806973 20356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:02.827221 20356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "eaf5d8b5893a4041a7f31da75f08ecd0"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "489016ee8415b3defe8a7163f8c7b1b3"
server_key_iv: "c95d78e0c2e378aecaa0bfcccd58146a"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.828027 20356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "eaf5d8b5893a4041a7f31da75f08ecd0"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "489016ee8415b3defe8a7163f8c7b1b3"
server_key_iv: "c95d78e0c2e378aecaa0bfcccd58146a"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.833346 20356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 05:25:02.846663 20371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.848168 20356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:25:02.848317 20356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "eaf5d8b5893a4041a7f31da75f08ecd0"
format_stamp: "Formatted at 2026-09-28 05:25:02 on dist-test-slave-515x"
server_key: "489016ee8415b3defe8a7163f8c7b1b3"
server_key_iv: "c95d78e0c2e378aecaa0bfcccd58146a"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:02.848511 20356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:02.874467 20356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:02.878422 20356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:02.878688 20356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:02.879638 20356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:02.880930 20356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:02.881004 20356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.881107 20356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:02.881156 20356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:02.912853 20356 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:42043
I20260928 05:25:02.912885 20483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:42043 every 8 connection(s)
I20260928 05:25:02.914142 20356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:25:02.921350 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 20356
I20260928 05:25:02.921494 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:25:02.921913 8691 external_mini_cluster.cc:1428] Setting key 62ba3cc4ae3f99f4d4a05b49d2ed9b99
Sep 28 05:25:02 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573102, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:02.933238 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.916426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:44473 (local address 127.8.124.254:33941)
0928 05:25:02.916856 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:02.916862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:02.917852 (+ 990us) server_negotiation.cc:407] Connection header received
0928 05:25:02.919501 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:02.919506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:02.919570 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:02.919663 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:02.921034 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.921826 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.923804 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.924234 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.924343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.927917 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:02.927922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:02.927925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:02.927963 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:02.930413 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.931066 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.931071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.931075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.931192 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.931591 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.931596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.931600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.931860 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:02.931982 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:02.932648 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 05:25:02.932877 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":182,"threads_started":1}
I20260928 05:25:02.934211 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.917001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33941 (local address 127.8.124.195:44473)
0928 05:25:02.917701 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:02.917737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:02.919245 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:02.919864 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:02.919873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:02.920264 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:02.920879 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.920896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.921971 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.921977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.923664 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.923678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.924475 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.924482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.924767 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.925386 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:02.925412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:02.927702 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:02.930539 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:02.930549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:02.930567 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:02.930958 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:02.931290 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:02.931302 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:02.931306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:02.931490 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:02.931977 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:02.931984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:02.932311 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 05:25:02.932551 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 05:25:02.933050 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
I20260928 05:25:02.935459 20484 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33941
I20260928 05:25:02.935747 20484 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:02.936573 20484 heartbeater.cc:511] Master 127.8.124.254:33941 requested a full tablet report, sending...
I20260928 05:25:02.937903 19994 ts_manager.cc:194] Registered new tserver with Master: eaf5d8b5893a4041a7f31da75f08ecd0 (127.8.124.195:42043)
I20260928 05:25:02.938642 19994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:44473
I20260928 05:25:02.949033 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:25:02.965485 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.953310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.8.124.254:33941)
0928 05:25:02.953466 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:02.953472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:02.953954 (+ 482us) server_negotiation.cc:407] Connection header received
0928 05:25:02.954135 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:02.954140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:02.954208 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:02.954284 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:02.955665 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.956390 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.957688 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.958092 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.958194 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.960437 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:02.960442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:02.960445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:02.960476 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:02.963124 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.963733 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.963738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.963741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.963866 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:02.964204 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:02.964209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:02.964213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:02.964455 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:02.964666 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:02.964951 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 05:25:02.965187 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:25:02.969070 19994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:25:02.976090 19994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:25:02.999189 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.988460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:33877 (local address 127.0.0.1:35320)
0928 05:25:02.989724 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:02.989764 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:02.989939 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:02.990955 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:02.990963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:02.990995 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:02.991397 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.991409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.993440 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.993446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.995861 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:02.995877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.998155 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.998162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:02.998405 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.998455 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 05:25:02.998600 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 05:25:02.998794 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":950,"threads_started":1}
I20260928 05:25:02.999226 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.988679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.8.124.194:33877)
0928 05:25:02.989228 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:02.989249 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:02.989861 (+ 612us) server_negotiation.cc:407] Connection header received
0928 05:25:02.990167 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:02.990175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:02.990457 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:02.990637 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:02.991575 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.993253 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.996054 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:02.997673 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:02.997815 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:02.998028 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:02.998669 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 05:25:02.998874 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":312,"threads_started":1}
I20260928 05:25:03.003525 20285 tablet_service.cc:1467] Processing CreateTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (DEFAULT_TABLE table=test-table [id=d69c401c00814db58ad80022bf5dc54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:03.005019 20285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:03.007243 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.988656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33687 (local address 127.0.0.1:46918)
0928 05:25:02.989724 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:02.989764 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:02.989928 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:03.000835 (+ 10907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:03.000841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:03.000868 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:03.001140 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.001150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.003049 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.003055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.004680 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.004691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.006442 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.006449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.006630 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.006662 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:25:03.006741 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:25:03.006882 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":43,"thread_start_us":955,"threads_started":1}
I20260928 05:25:03.007592 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.989275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.8.124.193:33687)
0928 05:25:02.999941 (+ 10666us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:02.999951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:02.999987 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:25:03.000113 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.000123 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.000463 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.000666 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:03.001344 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.002891 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.004828 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.006061 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.006183 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.007059 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.007130 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.007292 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10503,"thread_start_us":10343,"threads_started":1}
I20260928 05:25:03.009322 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.990180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42043 (local address 127.0.0.1:45804)
0928 05:25:02.991708 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:02.991728 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:02.991860 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:03.001661 (+ 9801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:03.001667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:03.001689 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:03.002055 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.002064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.003405 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.003410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.005434 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.005448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.007589 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.007595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.008450 (+ 855us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.008474 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:25:03.008535 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 05:25:03.008624 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":934,"threads_started":1}
I20260928 05:25:03.010030 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:02.990719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.8.124.195:42043)
0928 05:25:03.000806 (+ 10087us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.000814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.000859 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:25:03.000957 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.000964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.001180 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.001378 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:03.002206 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.003250 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.005584 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.007017 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.007146 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.009557 (+ 2411us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.009656 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.009790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9926,"thread_start_us":9765,"threads_started":1}
I20260928 05:25:03.011406 20142 tablet_service.cc:1467] Processing CreateTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (DEFAULT_TABLE table=test-table [id=d69c401c00814db58ad80022bf5dc54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:03.012988 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:03.013218 20419 tablet_service.cc:1467] Processing CreateTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (DEFAULT_TABLE table=test-table [id=d69c401c00814db58ad80022bf5dc54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:03.014330 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:03.020226 20503 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Bootstrap starting.
I20260928 05:25:03.023651 20503 tablet_bootstrap.cc:654] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:03.024852 20503 log.cc:824] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:03.025507 20504 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Bootstrap starting.
I20260928 05:25:03.027056 20506 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Bootstrap starting.
I20260928 05:25:03.028574 20503 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: No bootstrap required, opened a new log
I20260928 05:25:03.028854 20503 ts_tablet_manager.cc:1397] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 05:25:03.029361 20504 tablet_bootstrap.cc:654] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:03.030593 20504 log.cc:824] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:03.031342 20506 tablet_bootstrap.cc:654] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:03.032374 20506 log.cc:824] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:03.033676 20504 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: No bootstrap required, opened a new log
I20260928 05:25:03.033967 20504 ts_tablet_manager.cc:1397] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 05:25:03.034967 20506 tablet_bootstrap.cc:492] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: No bootstrap required, opened a new log
I20260928 05:25:03.035233 20506 ts_tablet_manager.cc:1397] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:25:03.039541 20503 raft_consensus.cc:348] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.040064 20503 raft_consensus.cc:374] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:03.040201 20503 raft_consensus.cc:729] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1f9b7146354b6180fd41d70c7ab222, State: Initialized, Role: FOLLOWER
I20260928 05:25:03.040961 20503 consensus_queue.cc:260] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.044070 20504 raft_consensus.cc:348] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.044567 20504 raft_consensus.cc:374] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:03.044699 20504 raft_consensus.cc:729] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf5d8b5893a4041a7f31da75f08ecd0, State: Initialized, Role: FOLLOWER
I20260928 05:25:03.045460 20504 consensus_queue.cc:260] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.045576 20506 raft_consensus.cc:348] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.046025 20506 raft_consensus.cc:374] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:03.046144 20506 raft_consensus.cc:729] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9419698b0ad4a508bcdb0a6442c8dbc, State: Initialized, Role: FOLLOWER
I20260928 05:25:03.046784 20506 consensus_queue.cc:260] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.049906 20503 ts_tablet_manager.cc:1428] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260928 05:25:03.050348 20350 heartbeater.cc:503] Master 127.8.124.254:33941 was elected leader, sending a full tablet report...
I20260928 05:25:03.057467 20484 heartbeater.cc:503] Master 127.8.124.254:33941 was elected leader, sending a full tablet report...
I20260928 05:25:03.058174 20504 ts_tablet_manager.cc:1428] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Time spent starting tablet: real 0.024s user 0.009s sys 0.014s
I20260928 05:25:03.058494 20506 ts_tablet_manager.cc:1428] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Time spent starting tablet: real 0.023s user 0.011s sys 0.009s
W20260928 05:25:03.165587 20485 tablet.cc:2369] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:25:03.181306 20511 raft_consensus.cc:482] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:25:03.181671 20511 raft_consensus.cc:504] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.183786 20511 leader_election.cc:288] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c1f9b7146354b6180fd41d70c7ab222 (127.8.124.194:33877), eaf5d8b5893a4041a7f31da75f08ecd0 (127.8.124.195:42043)
I20260928 05:25:03.191115 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.184339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:49567 (local address 127.8.124.195:42043)
0928 05:25:03.184526 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.184532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.184902 (+ 370us) server_negotiation.cc:407] Connection header received
0928 05:25:03.185213 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.185219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.185304 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.185377 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:03.186366 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.187383 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.189268 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.190239 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.190366 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.190417 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.190753 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.190867 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:25:03.191200 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.184068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42043 (local address 127.8.124.193:49567)
0928 05:25:03.184802 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:03.184821 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:03.185025 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:03.185629 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:03.185633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:03.185658 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:03.186199 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.186214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.187540 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.187545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.189129 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.189141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.190430 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.190437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.190596 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.190631 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:25:03.190731 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:25:03.190867 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":22,"spinlock_wait_cycles":126336,"thread_start_us":1991,"threads_started":1}
I20260928 05:25:03.191529 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.184711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:52117 (local address 127.8.124.194:33877)
0928 05:25:03.184882 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.184888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.185757 (+ 869us) server_negotiation.cc:407] Connection header received
0928 05:25:03.186096 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.186101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.186201 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.186373 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:03.186946 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.187925 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.189743 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.190655 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.190782 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.190846 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.191183 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.191312 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:25:03.191623 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.184118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:33877 (local address 127.8.124.193:52117)
0928 05:25:03.185638 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:03.185684 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:03.185913 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:03.186390 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:03.186395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:03.186420 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:03.186748 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.186761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.188086 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.188090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.189622 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:03.189637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.190857 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.190863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:03.191025 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.191053 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:25:03.191147 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:25:03.191263 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":3,"thread_start_us":784,"threads_started":1}
I20260928 05:25:03.192415 20439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd6d16c7a2d4dfabed6b90d6c78b5e7" candidate_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" is_pre_election: true
I20260928 05:25:03.192474 20305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd6d16c7a2d4dfabed6b90d6c78b5e7" candidate_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f9b7146354b6180fd41d70c7ab222" is_pre_election: true
I20260928 05:25:03.192855 20439 raft_consensus.cc:2454] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9419698b0ad4a508bcdb0a6442c8dbc in term 0.
I20260928 05:25:03.192869 20305 raft_consensus.cc:2454] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9419698b0ad4a508bcdb0a6442c8dbc in term 0.
I20260928 05:25:03.193507 20098 leader_election.cc:302] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9419698b0ad4a508bcdb0a6442c8dbc, eaf5d8b5893a4041a7f31da75f08ecd0; no voters:
I20260928 05:25:03.193890 20511 raft_consensus.cc:2781] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:25:03.194048 20511 raft_consensus.cc:482] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:25:03.194150 20511 raft_consensus.cc:3037] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:03.196610 20511 raft_consensus.cc:504] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.197166 20511 leader_election.cc:288] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [CANDIDATE]: Term 1 election: Requested vote from peers 9c1f9b7146354b6180fd41d70c7ab222 (127.8.124.194:33877), eaf5d8b5893a4041a7f31da75f08ecd0 (127.8.124.195:42043)
I20260928 05:25:03.197705 20439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd6d16c7a2d4dfabed6b90d6c78b5e7" candidate_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0"
I20260928 05:25:03.197757 20305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd6d16c7a2d4dfabed6b90d6c78b5e7" candidate_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f9b7146354b6180fd41d70c7ab222"
I20260928 05:25:03.197885 20439 raft_consensus.cc:3037] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:03.197953 20305 raft_consensus.cc:3037] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:03.199998 20439 raft_consensus.cc:2454] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9419698b0ad4a508bcdb0a6442c8dbc in term 1.
I20260928 05:25:03.200117 20305 raft_consensus.cc:2454] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9419698b0ad4a508bcdb0a6442c8dbc in term 1.
I20260928 05:25:03.200392 20098 leader_election.cc:302] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9419698b0ad4a508bcdb0a6442c8dbc, eaf5d8b5893a4041a7f31da75f08ecd0; no voters:
I20260928 05:25:03.200665 20511 raft_consensus.cc:2781] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:25:03.201081 20511 raft_consensus.cc:686] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 1 LEADER]: Becoming Leader. State: Replica: d9419698b0ad4a508bcdb0a6442c8dbc, State: Running, Role: LEADER
I20260928 05:25:03.201530 20511 consensus_queue.cc:237] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } }
I20260928 05:25:03.206173 19993 catalog_manager.cc:5495] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc reported cstate change: term changed from 0 to 1, leader changed from <none> to d9419698b0ad4a508bcdb0a6442c8dbc (127.8.124.193). New cstate: current_term: 1 leader_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9419698b0ad4a508bcdb0a6442c8dbc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 } health_report { overall_health: UNKNOWN } } }
I20260928 05:25:03.237103 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.231622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.8.124.193:33687)
0928 05:25:03.231830 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.231834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.231860 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:25:03.232025 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.232029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.232112 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.232234 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:25:03.232976 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.233716 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.235175 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.235581 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.235681 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.236125 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:25:03.236628 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:25:03.236756 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.236792 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.236856 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:25:03.244491 20439 raft_consensus.cc:1261] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 1 FOLLOWER]: Refusing update from remote peer d9419698b0ad4a508bcdb0a6442c8dbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:03.244491 20305 raft_consensus.cc:1261] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 1 FOLLOWER]: Refusing update from remote peer d9419698b0ad4a508bcdb0a6442c8dbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:03.245342 20511 consensus_queue.cc:1035] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1f9b7146354b6180fd41d70c7ab222" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:25:03.245656 20514 consensus_queue.cc:1035] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf5d8b5893a4041a7f31da75f08ecd0" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:25:03.257566 20519 mvcc.cc:205] Tried to move back new op lower bound from 7334187430880714752 to 7334187430717046784. Current Snapshot: MvccSnapshot[applied={T|T < 7334187430880714752}]
I20260928 05:25:03.277176 19993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:25:03.277494 19993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:25:03.283471 19993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec05c99dd733431c8f33e95fe3766c3a: Sending DeleteTablet for 3 replicas of tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7
I20260928 05:25:03.285200 20285 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:35320
I20260928 05:25:03.286537 20142 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:46918
I20260928 05:25:03.287853 20528 tablet_replica.cc:330] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: stopping tablet replica
I20260928 05:25:03.288239 20528 raft_consensus.cc:2229] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:25:03.288998 20419 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:03 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:45804
I20260928 05:25:03.289816 20529 tablet_replica.cc:330] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: stopping tablet replica
I20260928 05:25:03.290239 20529 raft_consensus.cc:2229] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:25:03.290709 20529 raft_consensus.cc:2258] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:03.291541 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 20074
I20260928 05:25:03.297837 20528 raft_consensus.cc:2258] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:03.300091 20528 ts_tablet_manager.cc:1905] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:03.306654 20528 ts_tablet_manager.cc:1918] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:25:03.306802 20528 log.cc:1196] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/bcd6d16c7a2d4dfabed6b90d6c78b5e7
I20260928 05:25:03.307418 20528 ts_tablet_manager.cc:1939] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P d9419698b0ad4a508bcdb0a6442c8dbc: Deleting consensus metadata
I20260928 05:25:03.307878 20527 tablet_replica.cc:330] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: stopping tablet replica
I20260928 05:25:03.308257 20527 raft_consensus.cc:2229] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:25:03.308727 20527 raft_consensus.cc:2258] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:03.309404 19981 catalog_manager.cc:4841] TS d9419698b0ad4a508bcdb0a6442c8dbc (127.8.124.193:33687): tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (table test-table [id=d69c401c00814db58ad80022bf5dc54c]) successfully deleted
I20260928 05:25:03.310535 20527 ts_tablet_manager.cc:1905] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:03.314739 20529 ts_tablet_manager.cc:1905] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:03.315797 20527 ts_tablet_manager.cc:1918] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:25:03.315937 20527 log.cc:1196] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/bcd6d16c7a2d4dfabed6b90d6c78b5e7
I20260928 05:25:03.316382 20527 ts_tablet_manager.cc:1939] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P 9c1f9b7146354b6180fd41d70c7ab222: Deleting consensus metadata
I20260928 05:25:03.318079 19980 catalog_manager.cc:4841] TS 9c1f9b7146354b6180fd41d70c7ab222 (127.8.124.194:33877): tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (table test-table [id=d69c401c00814db58ad80022bf5dc54c]) successfully deleted
I20260928 05:25:03.319450 20529 ts_tablet_manager.cc:1918] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:25:03.319595 20529 log.cc:1196] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/bcd6d16c7a2d4dfabed6b90d6c78b5e7
I20260928 05:25:03.319964 20529 ts_tablet_manager.cc:1939] T bcd6d16c7a2d4dfabed6b90d6c78b5e7 P eaf5d8b5893a4041a7f31da75f08ecd0: Deleting consensus metadata
I20260928 05:25:03.321240 19981 catalog_manager.cc:4841] TS eaf5d8b5893a4041a7f31da75f08ecd0 (127.8.124.195:42043): tablet bcd6d16c7a2d4dfabed6b90d6c78b5e7 (table test-table [id=d69c401c00814db58ad80022bf5dc54c]) successfully deleted
Sep 28 05:25:03 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:25:03.328698 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.292848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.8.124.193:33687)
0928 05:25:03.293233 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.293239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.307292 (+ 14053us) server_negotiation.cc:407] Connection header received
0928 05:25:03.307503 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.307508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.307613 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.307716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:03.309555 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.310760 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.312683 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.313575 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.313716 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.322756 (+ 9040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:03.322774 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:03.322788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:03.322828 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:03.326066 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.326714 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.326721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.326723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.326854 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.327220 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.327223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.327225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.327575 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:03.327831 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.328160 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.328396 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260928 05:25:03.329376 20202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:03.425015 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 20074
I20260928 05:25:03.439333 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 20213
Sep 28 05:25:03 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:25:03.452910 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.440071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.8.124.194:33877)
0928 05:25:03.440217 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.440221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.440238 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:25:03.440385 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.440388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.440451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.440529 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:03.441754 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.442493 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.443724 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.444078 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.444165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.448034 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:03.448047 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:03.448062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:03.448104 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:03.450562 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.451161 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.451167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.451170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.451293 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.451594 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.451598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.451601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.451980 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:03.452274 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.452400 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.452566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:25:03.453560 20345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:03.540766 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 20213
I20260928 05:25:03.553617 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 20356
Sep 28 05:25:03 dist-test-slave-515x krb5kdc[19518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:25:03.568063 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:03.554378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.8.124.195:42043)
0928 05:25:03.554531 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:03.554535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:03.554578 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:25:03.554729 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:03.554733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:03.554793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:03.554883 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:03.556158 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.556870 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.558182 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:03.558630 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:03.558762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:03.562751 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:03.562766 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:03.562781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:03.562817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:03.565359 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.566103 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.566110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.566113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.566234 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:03.566603 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:03.566607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:03.566609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:03.566960 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:03.567230 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:03.567544 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 05:25:03.567742 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:25:03.568604 20479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:03.656399 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 20356
I20260928 05:25:03.668879 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 19962
I20260928 05:25:03.669974 20024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:03.778064 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 19962
I20260928 05:25:03.798764 8691 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:25:04.133879 8691 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:25:04.134096 8691 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:25:04.469072 8691 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:25:04.469 UTC [19548] LOG: received smart shutdown request
2026-09-28 05:25:04.472 UTC [19548] LOG: background worker "logical replication launcher" (PID 19557) exited with exit code 1
2026-09-28 05:25:04.472 UTC [19552] LOG: shutting down
2026-09-28 05:25:04.625 UTC [19548] LOG: database system is shut down
2026-09-28T05:25:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47445 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:25:04 dist-test-slave-515x krb5kdc[20543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:25:04 dist-test-slave-515x krb5kdc[20543](info): set up 2 sockets
Sep 28 05:25:04 dist-test-slave-515x krb5kdc[20543](info): commencing operation
krb5kdc: starting...
W20260928 05:25:05.733997 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:25:05 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:25:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangeradmin_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangeradmin_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangerlookup_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangerlookup_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
I20260928 05:25:06.020898 8691 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/postgres -l logfile start
2026-09-28 05:25:08.804 UTC [20573] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:25:08.804 UTC [20573] LOG: listening on IPv4 address "127.8.124.212", port 43833
2026-09-28 05:25:08.807 UTC [20573] LOG: listening on Unix socket "/tmp/.s.PGSQL.43833"
2026-09-28 05:25:08.827 UTC [20576] LOG: database system was shut down at 2026-09-28 05:25:07 UTC
2026-09-28 05:25:08.830 UTC [20573] LOG: database system is ready to accept connections
I20260928 05:25:09.729491 8691 mini_postgres.cc:96] Postgres bound to 43833
2026-09-28 05:25:09.745 UTC [20584] FATAL: database "slave" does not exist
127.8.124.212:43833 - accepting connections
I20260928 05:25:09.745591 8691 mini_ranger.cc:161] Starting Ranger...
I20260928 05:25:09.768985 8691 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:25:10.196336 8691 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:25:10.196446 8691 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin
2026-09-28 05:25:10,470 [I] DB FLAVOR :POSTGRES
2026-09-28 05:25:10,471 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:25:10,471 [I] Checking connection..
2026-09-28 05:25:10,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:25:10,717 [I] Checking connection passed.
2026-09-28 05:25:10,717 [I] --------- Verifying version history table ---------
2026-09-28 05:25:10,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:25:10,970 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:25:10,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:25:11,216 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:25:11,216 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:25:11,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:25:11.434 UTC [20648] WARNING: there is no transaction in progress
2026-09-28 05:25:11,439 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:25:11,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:25:11,687 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:25:11,687 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:25:11,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:25:11,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:25:12,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-515x.c.gcp-upstream.internal','N') ;"
2026-09-28 05:25:12,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:25:12,613 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:25:12,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:25:12,861 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:25:12,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:25:13,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:25:13,316 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:25:13,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:25:14.211 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.226 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.242 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.257 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.272 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.288 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.353 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.359 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.373 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.382 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.392 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.405 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.415 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.422 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.428 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:14.435 UTC [20756] WARNING: there is no transaction in progress
2026-09-28 05:25:15,006 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:25:15,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:25:15,243 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:25:15,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:25:15,478 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:25:15,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:25:15,706 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:25:15,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:25:15,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:25:16,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-515x.c.gcp-upstream.internal';"
2026-09-28 05:25:16,415 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:25:16,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:25:16,663 [I] DB_PATCHES have already been applied
I20260928 05:25:16.668711 8691 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbMG2Xa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:25:16.668802 8691 mini_ranger.cc:193] Using host: 127.8.124.212
I20260928 05:25:16.682364 8691 mini_ranger.cc:227] Ranger admin URL: http://127.8.124.212:58835
Sep 28, 2026 5:25:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:25:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:25:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:25:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:25:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:25:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:25:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:25:17.711163 8691 mini_ranger.cc:160] Time spent starting Ranger: real 7.966s user 0.000s sys 0.035s
Sep 28, 2026 5:25:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:25:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 5:25:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:25:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:25:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:25:39.023252 8691 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.124.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.124.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangerkms_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.124.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/rangerkms_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.124.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.124.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.124.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/HTTP_127.8.124.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:25:39 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573139, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:25:39.175297 8691 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:25:39.196954 8691 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:25:40.170058 8691 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:25:40.170176 8691 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms
I20260928 05:25:40.170215 8691 mini_ranger_kms.cc:227] Using postgres at 127.8.124.212:43833
2026-09-28 05:25:40,491 [I] DB FLAVOR :POSTGRES
2026-09-28 05:25:40,492 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:25:40,492 [I] Checking connection
2026-09-28 05:25:40,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:25:40,722 [I] connection success
2026-09-28 05:25:40,722 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:25:40,722 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:25:40,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:25:40,942 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:25:40,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:25:40,942 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:25:40,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.124.212:43833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:25:41,181 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:25:41.616768 8691 mini_ranger_kms.cc:310] Created kms service
I20260928 05:25:41.786080 8691 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:25:41.828733 8691 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:25:41 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573139, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:42.211194 8691 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:25:42.211498 8691 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbMG2Xa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbMG2Xa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:25:42.211561 8691 mini_ranger_kms.cc:240] Using host: 127.8.124.212
I20260928 05:25:42.225219 8691 mini_ranger_kms.cc:276] Ranger KMS PID: 20936
I20260928 05:25:42.225342 8691 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.124.212:59031
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:25:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 05:25:44.289119 8691 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.114s user 0.077s sys 0.254s
I20260928 05:25:44.289361 8691 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:25:44.289515 8691 mini_ranger_kms.cc:401] 127.8.124.212:59031/kms/v1/keys
Sep 28 05:25:44 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790573144, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 05:25:44 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573144, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Sep 28 05:25:45 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573144, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.124.212@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Sep 28 05:25:47 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:25:47.431459 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33045
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33045
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:25:47.741710 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:47.741968 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:47.742090 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:47.749027 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:25:47.749133 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:47.749205 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:47.749271 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:25:47.749331 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:25:47.759194 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.124.212:59031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33045
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:47.760644 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:47.762097 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:47.773059 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:47.773083 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:47.773146 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:47.773342 20987 server_base.cc:1034] running on GCE node
I20260928 05:25:47.774289 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:47.775538 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:47.776718 20987 hybrid_clock.cc:648] HybridClock initialized: now 1790573147776677 us; error 35 us; skew 500 ppm
I20260928 05:25:47.777011 20987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:47 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:47.782635 20987 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:25:47.783922 20987 webserver.cc:460] Webserver started at http://127.8.124.254:35351/ using document root <none> and password file <none>
I20260928 05:25:47.784492 20987 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:47.784576 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:47.784808 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:47 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:47.832037 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "c3f4466e71274415854a2bedc911ccd8"
format_stamp: "Formatted at 2026-09-28 05:25:47 on dist-test-slave-515x"
server_key: "2e7cf41ce22b68804bfb576cac6b1cf3"
server_key_iv: "6eeb62333ea4ed139e723f4a0f3d603b"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:47.833138 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "c3f4466e71274415854a2bedc911ccd8"
format_stamp: "Formatted at 2026-09-28 05:25:47 on dist-test-slave-515x"
server_key: "2e7cf41ce22b68804bfb576cac6b1cf3"
server_key_iv: "6eeb62333ea4ed139e723f4a0f3d603b"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:47.839936 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 05:25:47.858232 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:47.860116 20987 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:25:47.860306 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "c3f4466e71274415854a2bedc911ccd8"
format_stamp: "Formatted at 2026-09-28 05:25:47 on dist-test-slave-515x"
server_key: "2e7cf41ce22b68804bfb576cac6b1cf3"
server_key_iv: "6eeb62333ea4ed139e723f4a0f3d603b"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:47.860610 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:47.876387 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:47.879972 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:47.880236 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:47.897683 20987 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:33045
I20260928 05:25:47.897683 21053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:33045 every 8 connection(s)
I20260928 05:25:47.898952 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:25:47.900581 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 20987
I20260928 05:25:47.900790 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:25:47.901618 8691 external_mini_cluster.cc:1428] Setting key 0456de36c80142aa61d17d46864136d9
I20260928 05:25:47.902885 21054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:47.917507 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Bootstrap starting.
I20260928 05:25:47.921500 21054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:47.922700 21054 log.cc:824] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:47.925338 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: No bootstrap required, opened a new log
Sep 28 05:25:47 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:47.933506 21054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:47.934029 21054 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:47.934175 21054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f4466e71274415854a2bedc911ccd8, State: Initialized, Role: FOLLOWER
I20260928 05:25:47.934947 21054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:47.935230 21054 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:25:47.935361 21054 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:25:47.935547 21054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:47.938464 21054 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:47.939081 21054 leader_election.cc:302] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f4466e71274415854a2bedc911ccd8; no voters:
I20260928 05:25:47.939731 21054 leader_election.cc:288] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:25:47.939858 21059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:25:47.943274 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:47.903674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.8.124.254:33045)
0928 05:25:47.918225 (+ 14551us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:47.918242 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:47.918292 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:25:47.920610 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:47.920639 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:47.921110 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:47.921563 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:47.923254 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:47.925347 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:47.927345 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:47.928469 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:47.928651 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:47.933433 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:47.933443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:47.933463 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:47.933518 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:47.937325 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:47.938276 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:47.938285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:47.938293 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:47.938492 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:47.938905 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:47.938910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:47.938913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:47.939647 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:47.939951 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:47.941351 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0928 05:25:47.941812 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14300,"thread_start_us":14100,"threads_started":1}
I20260928 05:25:47.946976 21059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 LEADER]: Becoming Leader. State: Replica: c3f4466e71274415854a2bedc911ccd8, State: Running, Role: LEADER
I20260928 05:25:47.947633 21054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:25:47.947571 21059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:47.950461 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f4466e71274415854a2bedc911ccd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } } }
I20260928 05:25:47.950865 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:47.953637 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f4466e71274415854a2bedc911ccd8. Latest consensus state: current_term: 1 leader_uuid: "c3f4466e71274415854a2bedc911ccd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } } }
I20260928 05:25:47.954402 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:47.956832 21066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:25:47.961248 21066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:25:47.962016 21054 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:25:47.978366 21066 catalog_manager.cc:1347] Generated new cluster ID: fe8ee65b03c44a9286e73417c221457a
I20260928 05:25:47.978557 21066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:25:47.995121 21066 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:25:47.997159 21066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:25:48.006979 21066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Generated new TSK 0
I20260928 05:25:48.008100 21066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:25:48.711479 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:25:49.140933 21099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:49.141234 21099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:49.141403 21099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:49.151691 21099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:49.151896 21099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:49.152112 21099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:25:49.168102 21099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:49.169989 21099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:49.171697 21099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:49.198185 21110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:49.198841 21112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:49.200317 21109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:49.200312 21099 server_base.cc:1034] running on GCE node
I20260928 05:25:49.201066 21099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:49.202121 21099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:49.203397 21099 hybrid_clock.cc:648] HybridClock initialized: now 1790573149203334 us; error 63 us; skew 500 ppm
I20260928 05:25:49.203715 21099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:49.206780 21099 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:25:49.213919 21099 webserver.cc:460] Webserver started at http://127.8.124.193:45177/ using document root <none> and password file <none>
I20260928 05:25:49.214615 21099 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:49.214735 21099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:49.215029 21099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:49.247231 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "5af08b01ce214a33b377b8965ae115bc"
format_stamp: "Formatted at 2026-09-28 05:25:49 on dist-test-slave-515x"
server_key: "9476f58930e6a313424b2781ee870386"
server_key_iv: "7fd9adbee4fa22315bdcdc415b8297a9"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:49.249428 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "5af08b01ce214a33b377b8965ae115bc"
format_stamp: "Formatted at 2026-09-28 05:25:49 on dist-test-slave-515x"
server_key: "9476f58930e6a313424b2781ee870386"
server_key_iv: "7fd9adbee4fa22315bdcdc415b8297a9"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:49.257627 21099 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.002s
I20260928 05:25:49.275820 21119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:49.277742 21099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 05:25:49.277988 21099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "5af08b01ce214a33b377b8965ae115bc"
format_stamp: "Formatted at 2026-09-28 05:25:49 on dist-test-slave-515x"
server_key: "9476f58930e6a313424b2781ee870386"
server_key_iv: "7fd9adbee4fa22315bdcdc415b8297a9"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:49.278321 21099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:49.319273 21099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:49.324956 21099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:49.325351 21099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:49.326990 21099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:49.328810 21099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:49.328971 21099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:49.329170 21099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:49.329288 21099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:49.387817 21099 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33763
I20260928 05:25:49.388047 21231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33763 every 8 connection(s)
I20260928 05:25:49.389474 21099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:25:49.392542 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21099
I20260928 05:25:49.392736 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:25:49.393554 8691 external_mini_cluster.cc:1428] Setting key be5cdfa31acc893968610dabc4ad29ac
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:25:49.445714 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:49.395483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:57133 (local address 127.8.124.254:33045)
0928 05:25:49.409410 (+ 13927us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:49.409417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:49.423716 (+ 14299us) server_negotiation.cc:407] Connection header received
0928 05:25:49.423793 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:49.423826 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:49.423926 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:49.424031 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:49.427529 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:49.428592 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:49.431288 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:49.431993 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:49.432093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:49.438908 (+ 6815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:49.438914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:49.438918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:49.438959 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:49.442032 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:49.442921 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:49.442928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:49.442932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:49.443084 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:49.443576 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:49.443582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:49.443585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:49.443922 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:49.444037 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:49.445074 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 05:25:49.445301 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13770,"thread_start_us":13937,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:25:49.446818 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:49.394943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.193:57133)
0928 05:25:49.420735 (+ 25792us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:49.420826 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:49.423276 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:49.425814 (+ 2538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:49.425828 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:49.426458 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:49.427311 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:49.427338 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:49.428788 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:49.428793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:49.431112 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:49.431128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:49.432461 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:49.432469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:49.432762 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:49.433859 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:49.433888 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:49.438621 (+ 4733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:49.442211 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:49.442220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:49.442238 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:49.442752 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:49.443198 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:49.443202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:49.443205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:49.443455 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:49.444126 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:49.444133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:49.444622 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 05:25:49.444919 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 05:25:49.445497 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25457,"thread_start_us":25228,"threads_started":1}
I20260928 05:25:49.449061 21232 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:49.449364 21232 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:49.450162 21232 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:49.452435 21019 ts_manager.cc:194] Registered new tserver with Master: 5af08b01ce214a33b377b8965ae115bc (127.8.124.193:33763)
I20260928 05:25:49.454331 21019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:57133
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:25:49.486614 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
W20260928 05:25:49.928386 21238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:49.928857 21238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:49.929039 21238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:49.940174 21238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:49.940398 21238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:49.940620 21238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:25:49.955644 21238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:49.957271 21238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:49.958945 21238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:49.984207 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:49.986263 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:49.990630 21238 server_base.cc:1034] running on GCE node
W20260928 05:25:49.991408 21250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:49.992162 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:49.993155 21238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:49.994380 21238 hybrid_clock.cc:648] HybridClock initialized: now 1790573149994312 us; error 60 us; skew 500 ppm
I20260928 05:25:49.994751 21238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:49.998047 21238 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:25:49.999697 21238 webserver.cc:460] Webserver started at http://127.8.124.194:44835/ using document root <none> and password file <none>
I20260928 05:25:50.000588 21238 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:50.000717 21238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:50.001034 21238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:49 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:50.019203 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "6e059157282c4fd7bbfd5beedd673cb1"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "69f396ff679903e854efd70e283f8d9a"
server_key_iv: "73c6aef2822adf4885b38c43bf1b9e21"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.020146 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "6e059157282c4fd7bbfd5beedd673cb1"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "69f396ff679903e854efd70e283f8d9a"
server_key_iv: "73c6aef2822adf4885b38c43bf1b9e21"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.027168 21238 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260928 05:25:50.057135 21262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.058794 21238 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.010s sys 0.004s
I20260928 05:25:50.058959 21238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "6e059157282c4fd7bbfd5beedd673cb1"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "69f396ff679903e854efd70e283f8d9a"
server_key_iv: "73c6aef2822adf4885b38c43bf1b9e21"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.059167 21238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:50.079408 21238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:50.083293 21238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:50.083559 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:50.084514 21238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:50.085891 21238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:50.085969 21238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.086089 21238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:50.086141 21238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.128227 21238 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45195
I20260928 05:25:50.128301 21374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45195 every 8 connection(s)
I20260928 05:25:50.129470 21238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:25:50.131752 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21238
I20260928 05:25:50.131920 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:25:50.132507 8691 external_mini_cluster.cc:1428] Setting key 43d9bcd54db329c27ec5fd240215a7b0
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:50.151942 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.132239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:41123 (local address 127.8.124.254:33045)
0928 05:25:50.133856 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:50.133864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:50.134546 (+ 682us) server_negotiation.cc:407] Connection header received
0928 05:25:50.136967 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:50.136973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:50.137069 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:50.137173 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:50.139104 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.140135 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.142258 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.142857 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.142974 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.146382 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:50.146386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:50.146389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:50.146422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:50.149024 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.149710 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.149716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.149719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.149838 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.150275 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.150279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.150281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.150552 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:50.150677 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:50.151364 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 05:25:50.151589 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":1302,"threads_started":1}
I20260928 05:25:50.153033 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.132904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.194:41123)
0928 05:25:50.134394 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:50.134445 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:50.136643 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:50.137544 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:50.137556 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:50.138123 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:50.138909 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:50.138934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.140296 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.140302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:50.142101 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:50.142115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.143117 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.143124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:50.143427 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.144149 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:50.144176 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:50.146163 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:50.149159 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:50.149168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:50.149186 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:50.149581 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:50.149937 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:50.149942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:50.149946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:50.150160 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:50.150668 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:50.150675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:50.151005 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 05:25:50.151228 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 05:25:50.151728 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":1016,"threads_started":1}
I20260928 05:25:50.154374 21375 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:50.154663 21375 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:50.155336 21375 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:50.156677 21019 ts_manager.cc:194] Registered new tserver with Master: 6e059157282c4fd7bbfd5beedd673cb1 (127.8.124.194:45195)
I20260928 05:25:50.157519 21019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:41123
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:25:50.189105 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:25:50.457861 21232 heartbeater.cc:503] Master 127.8.124.254:33045 was elected leader, sending a full tablet report...
W20260928 05:25:50.523129 21381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:50.523378 21381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:50.523488 21381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:50.530512 21381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:50.530612 21381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:50.530820 21381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:25:50.540289 21381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:50.541529 21381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:50.542783 21381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:50.554862 21387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:50.554849 21386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:50.555455 21389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:50.555804 21381 server_base.cc:1034] running on GCE node
I20260928 05:25:50.556348 21381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:50.557305 21381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:50.558493 21381 hybrid_clock.cc:648] HybridClock initialized: now 1790573150558442 us; error 45 us; skew 500 ppm
I20260928 05:25:50.558781 21381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:50.561887 21381 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:25:50.563335 21381 webserver.cc:460] Webserver started at http://127.8.124.195:38157/ using document root <none> and password file <none>
I20260928 05:25:50.563985 21381 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:50.564091 21381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:50.564369 21381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:50.578955 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "eef02ead408d471f9bccd6d601439ffc"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "d82009b3b5e455b23d98f4cda8f54b3f"
server_key_iv: "f5333b6d030c410a88dbcba113e5f073"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.579970 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "eef02ead408d471f9bccd6d601439ffc"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "d82009b3b5e455b23d98f4cda8f54b3f"
server_key_iv: "f5333b6d030c410a88dbcba113e5f073"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.586092 21381 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 05:25:50.602095 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.603576 21381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:25:50.603729 21381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "eef02ead408d471f9bccd6d601439ffc"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "d82009b3b5e455b23d98f4cda8f54b3f"
server_key_iv: "f5333b6d030c410a88dbcba113e5f073"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:50.603940 21381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:50.627432 21381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:50.631268 21381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:50.631531 21381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:50.632467 21381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:50.633880 21381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:50.633952 21381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.634063 21381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:50.634111 21381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:50.668479 21381 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:33919
I20260928 05:25:50.668497 21508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:33919 every 8 connection(s)
I20260928 05:25:50.669804 21381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:25:50.673717 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21381
I20260928 05:25:50.673874 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:25:50.674412 8691 external_mini_cluster.cc:1428] Setting key f20a23999fce7f9817b2dee782df6115
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:50.690652 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.672325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:57099 (local address 127.8.124.254:33045)
0928 05:25:50.672867 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:50.672873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:50.675016 (+ 2143us) server_negotiation.cc:407] Connection header received
0928 05:25:50.676024 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:50.676030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:50.676129 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:50.676219 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:50.677798 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.678930 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.680469 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.681107 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.681242 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.684494 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:50.684501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:50.684506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:50.684545 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:50.687613 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.688304 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.688309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.688313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.688466 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.688938 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.688942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.688944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.689192 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:50.689343 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:50.690080 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 05:25:50.690300 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":275,"threads_started":1}
I20260928 05:25:50.691605 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.672909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.195:57099)
0928 05:25:50.673824 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:50.673878 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:50.675723 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:50.676491 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:50.676504 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:50.676947 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:50.677588 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:50.677644 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.679101 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.679105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:50.680330 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:50.680341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.681289 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.681294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:50.681535 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.682175 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:50.682202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:50.684290 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:50.687732 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:50.687739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:50.687751 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:50.688173 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:50.688573 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:50.688578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:50.688581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:50.688830 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:50.689319 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:50.689327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:50.689714 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 05:25:50.689994 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 05:25:50.690495 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":415,"threads_started":1}
I20260928 05:25:50.692866 21509 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:50.693123 21509 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:50.693893 21509 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:50.695613 21019 ts_manager.cc:194] Registered new tserver with Master: eef02ead408d471f9bccd6d601439ffc (127.8.124.195:33919)
I20260928 05:25:50.696723 21019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:57099
I20260928 05:25:50.702256 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:25:50.703944 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21099
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:25:50.723062 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.704831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.8.124.193:33763)
0928 05:25:50.705513 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:50.705523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:50.705554 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:25:50.705659 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:50.705670 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:50.705957 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:50.706127 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:50.707577 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.709102 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.710637 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.711554 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.711697 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.716183 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:50.716203 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:50.716236 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:50.716291 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:50.720008 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.720727 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.720737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.720741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.720925 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.721239 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.721244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.721248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.721779 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:50.722081 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:50.722463 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 05:25:50.722742 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":327,"threads_started":1}
I20260928 05:25:50.724267 21227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:50.808390 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21099
I20260928 05:25:50.820434 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21238
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:25:50.835012 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.821262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.8.124.194:45195)
0928 05:25:50.821820 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:50.821827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:50.821852 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:25:50.821910 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:50.821916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:50.822122 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:50.822242 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:50.823396 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.824393 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.825552 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.826137 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.826238 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.830143 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:50.830156 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:50.830171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:50.830207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:50.832586 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.833176 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.833182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.833185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.833302 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.833591 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.833594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.833708 (+ 114us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.834022 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:50.834244 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:50.834479 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 05:25:50.834684 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":278,"threads_started":1}
I20260928 05:25:50.836208 21370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:50.916237 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21238
I20260928 05:25:50.928428 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21381
Sep 28 05:25:50 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:25:50.943917 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:50.929436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.8.124.195:33919)
0928 05:25:50.930042 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:50.930049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:50.930076 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:25:50.930151 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:50.930157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:50.930371 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:50.930501 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:50.931755 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.932738 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.934100 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:50.934668 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:50.934761 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:50.938908 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:50.938919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:50.938934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:50.938967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:50.941432 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.942181 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.942189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.942191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.942314 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:50.942636 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:50.942639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:50.942641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:50.942944 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:50.943182 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:50.943387 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 05:25:50.943566 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":317,"threads_started":1}
I20260928 05:25:50.945134 21504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:51.024214 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21381
I20260928 05:25:51.036079 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 20987
I20260928 05:25:51.037271 21049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:51.139708 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 20987
I20260928 05:25:51.156603 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33045
--webserver_interface=127.8.124.254
--webserver_port=35351
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33045
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:25:51.463272 21521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:51.463546 21521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:51.463663 21521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:51.470693 21521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:25:51.470794 21521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:51.470851 21521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:51.470925 21521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:25:51.470984 21521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:25:51.480770 21521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33045
--ranger_config_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=35351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:51.482081 21521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:51.483402 21521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:51.494230 21529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:51.494191 21527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:51.494285 21526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:51.495086 21521 server_base.cc:1034] running on GCE node
I20260928 05:25:51.495810 21521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:51.497165 21521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:51.498381 21521 hybrid_clock.cc:648] HybridClock initialized: now 1790573151498325 us; error 41 us; skew 500 ppm
I20260928 05:25:51.498750 21521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:51 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573151, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:51.501545 21521 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:25:51.502923 21521 webserver.cc:460] Webserver started at http://127.8.124.254:35351/ using document root <none> and password file <none>
I20260928 05:25:51.503533 21521 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:51.503644 21521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:25:51 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573151, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:51.571403 21521 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.008s
I20260928 05:25:51.575004 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:51.576576 21521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:25:51.576756 21521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "c3f4466e71274415854a2bedc911ccd8"
format_stamp: "Formatted at 2026-09-28 05:25:47 on dist-test-slave-515x"
server_key: "2e7cf41ce22b68804bfb576cac6b1cf3"
server_key_iv: "6eeb62333ea4ed139e723f4a0f3d603b"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:51.578065 21521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:51.592022 21521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:51.595324 21521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:51.595535 21521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:51.612447 21521 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:33045
I20260928 05:25:51.612475 21588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:33045 every 8 connection(s)
I20260928 05:25:51.613827 21521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:25:51.616814 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21521
I20260928 05:25:51.617847 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:33763
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:33045
--webserver_port=45177
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:25:51.621961 21589 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:25:51.626075 21589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Bootstrap starting.
I20260928 05:25:51.644207 21589 log.cc:824] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:51.652981 21589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:25:51.653496 21589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Bootstrap complete.
I20260928 05:25:51.661988 21589 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:51.662792 21589 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f4466e71274415854a2bedc911ccd8, State: Initialized, Role: FOLLOWER
I20260928 05:25:51.663395 21589 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:51.663601 21589 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:25:51.663738 21589 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:25:51.663885 21589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:25:51.667603 21589 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:51.668068 21589 leader_election.cc:302] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f4466e71274415854a2bedc911ccd8; no voters:
I20260928 05:25:51.668617 21589 leader_election.cc:288] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:25:51.668777 21593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:25:51.669782 21593 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [term 2 LEADER]: Becoming Leader. State: Replica: c3f4466e71274415854a2bedc911ccd8, State: Running, Role: LEADER
I20260928 05:25:51.670403 21593 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } }
I20260928 05:25:51.670702 21589 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:25:51.672681 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f4466e71274415854a2bedc911ccd8. Latest consensus state: current_term: 2 leader_uuid: "c3f4466e71274415854a2bedc911ccd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } } }
I20260928 05:25:51.673038 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:51.673516 21594 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3f4466e71274415854a2bedc911ccd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4466e71274415854a2bedc911ccd8" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33045 } } }
I20260928 05:25:51.673823 21594 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:51.684374 21600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:25:51.690231 21600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:25:51.691593 21600 catalog_manager.cc:1259] Loaded cluster ID: fe8ee65b03c44a9286e73417c221457a
I20260928 05:25:51.691752 21600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:25:51.695789 21600 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:25:51.697484 21600 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Loaded TSK: 0
I20260928 05:25:51.699084 21600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:25:51.701422 21589 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 05:25:52.047675 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:52.048224 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:52.048422 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:52.059543 21591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:52.059757 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:52.060002 21591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:25:52.075516 21591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:33763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=45177
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:52.077504 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:52.079329 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:52.102499 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:52.104139 21627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:52.112363 21591 server_base.cc:1034] running on GCE node
W20260928 05:25:52.112654 21629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:52.113485 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:52.116291 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:52.117588 21591 hybrid_clock.cc:648] HybridClock initialized: now 1790573152117514 us; error 65 us; skew 500 ppm
I20260928 05:25:52.118009 21591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:52.122241 21591 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:25:52.127130 21591 webserver.cc:460] Webserver started at http://127.8.124.193:45177/ using document root <none> and password file <none>
I20260928 05:25:52.128131 21591 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:52.128358 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:52.165972 21591 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 05:25:52.169842 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:52.171976 21591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:25:52.172149 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "5af08b01ce214a33b377b8965ae115bc"
format_stamp: "Formatted at 2026-09-28 05:25:49 on dist-test-slave-515x"
server_key: "9476f58930e6a313424b2781ee870386"
server_key_iv: "7fd9adbee4fa22315bdcdc415b8297a9"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:52.173498 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:52.195043 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:52.198870 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:52.199133 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:52.200127 21591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:52.201646 21591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:52.201741 21591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:52.201840 21591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:52.201905 21591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:52.270562 21591 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:33763
I20260928 05:25:52.270612 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:33763 every 8 connection(s)
I20260928 05:25:52.272032 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:25:52.278802 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21591
I20260928 05:25:52.279587 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:45195
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:33045
--webserver_port=44835
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:52.313073 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:52.275139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.193:59495)
0928 05:25:52.289166 (+ 14027us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:52.289227 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:52.291783 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:52.293311 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:52.293323 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:52.293896 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:52.294433 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:52.294451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.296212 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.296218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:52.297853 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:52.297867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.299597 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.299606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:52.299897 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:52.300647 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:52.300683 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:52.303189 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:52.307783 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:52.307793 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:52.307812 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:52.308442 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:52.308978 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:52.308983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:52.308987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:52.309257 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:52.310405 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:52.310412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:52.310828 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 05:25:52.311150 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 05:25:52.311710 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13678,"thread_start_us":14394,"threads_started":1}
I20260928 05:25:52.313061 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:52.275120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:59495 (local address 127.8.124.254:33045)
0928 05:25:52.275865 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:52.275884 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:52.290673 (+ 14789us) server_negotiation.cc:407] Connection header received
0928 05:25:52.292399 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:52.292414 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:52.292778 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:52.293169 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:52.294628 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.296067 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.298017 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.299036 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.299215 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:52.303486 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:52.303496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:52.303516 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:52.303561 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:52.307617 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:52.308630 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:52.308640 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:52.308648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:52.308836 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:52.309399 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:52.309404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:52.309408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:52.310129 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:52.310330 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:52.311408 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0928 05:25:52.311821 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":261,"threads_started":1}
I20260928 05:25:52.317205 21751 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:52.317514 21751 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:52.318354 21751 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
W20260928 05:25:52.321481 21751 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:33045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:25:52.321784 21751 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:52.322098 21751 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:52.323370 21751 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:52.323700 21751 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
W20260928 05:25:52.324800 21751 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:25:52.674311 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:52.674588 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:52.674831 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:52.686264 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:52.686399 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:52.686587 21755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:25:52.701749 21755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:45195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=44835
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:52.703542 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:52.705283 21755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:52.719669 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:52.719642 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:52.719741 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:52.721201 21755 server_base.cc:1034] running on GCE node
I20260928 05:25:52.721905 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:52.722977 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:52.724215 21755 hybrid_clock.cc:648] HybridClock initialized: now 1790573152724160 us; error 46 us; skew 500 ppm
I20260928 05:25:52.724506 21755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:52.727445 21755 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:25:52.732722 21755 webserver.cc:460] Webserver started at http://127.8.124.194:44835/ using document root <none> and password file <none>
I20260928 05:25:52.733438 21755 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:52.733557 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:52.774348 21755 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.004s
I20260928 05:25:52.784791 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:52.786978 21755 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260928 05:25:52.787227 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "6e059157282c4fd7bbfd5beedd673cb1"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "69f396ff679903e854efd70e283f8d9a"
server_key_iv: "73c6aef2822adf4885b38c43bf1b9e21"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:52.788735 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:52.816380 21755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:52.821451 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:52.821796 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:52.828084 21755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:52.830078 21755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:52.830212 21755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:52.830331 21755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:52.830410 21755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:52.950217 21755 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45195
I20260928 05:25:52.950273 21895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45195 every 8 connection(s)
I20260928 05:25:52.952097 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:25:52.962160 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21755
I20260928 05:25:52.962885 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.124.212:59031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:33919
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:33045
--webserver_port=38157
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:25:52 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:52.978498 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:52.955414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.194:46697)
0928 05:25:52.958137 (+ 2723us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:52.958189 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:52.959798 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:52.960586 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:52.960594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:52.961076 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:52.961695 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:52.961719 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.963193 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.963198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:52.965031 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:52.965045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.966224 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.966231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:52.966501 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:52.967260 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:52.967288 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:52.969907 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:52.973780 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:52.973789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:52.973808 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:52.974353 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:52.974771 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:52.974776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:52.974779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:52.975019 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:52.975679 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:52.975687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:52.976103 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 05:25:52.976385 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 05:25:52.976938 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2463,"thread_start_us":2259,"threads_started":1}
I20260928 05:25:52.979564 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:52.954793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:46697 (local address 127.8.124.254:33045)
0928 05:25:52.955244 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:52.955251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:52.960065 (+ 4814us) server_negotiation.cc:407] Connection header received
0928 05:25:52.960125 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:52.960128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:52.960196 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:52.960362 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:52.961912 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.963000 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.965218 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:52.965825 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:52.965938 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:52.970191 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:52.970204 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:52.970209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:52.970251 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:52.973570 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:52.974494 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:52.974499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:52.974503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:52.974655 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:52.975135 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:52.975139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:52.975142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:52.975474 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:52.975605 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:52.979038 (+ 3433us) server_negotiation.cc:299] Negotiation successful
0928 05:25:52.979266 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":195,"threads_started":1}
I20260928 05:25:52.981026 21896 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:52.981319 21896 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:52.982223 21896 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:52.984157 21554 ts_manager.cc:194] Registered new tserver with Master: 6e059157282c4fd7bbfd5beedd673cb1 (127.8.124.194:45195)
I20260928 05:25:52.986266 21554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:46697
Sep 28 05:25:53 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:53.324949 21751 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:53.325382 21751 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:53.326692 21554 ts_manager.cc:194] Registered new tserver with Master: 5af08b01ce214a33b377b8965ae115bc (127.8.124.193:33763)
I20260928 05:25:53.327819 21554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:59495
W20260928 05:25:53.405041 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:53.405349 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:53.405484 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:53.412914 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:53.413028 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:53.413180 21900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:25:53.423864 21900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.124.212:59031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:33919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=38157
--tserver_master_addrs=127.8.124.254:33045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:53.425437 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:53.427089 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:53.442482 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:53.443184 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:53.447868 21900 server_base.cc:1034] running on GCE node
W20260928 05:25:53.449877 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:53.450690 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:53.451970 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:53.453243 21900 hybrid_clock.cc:648] HybridClock initialized: now 1790573153453182 us; error 47 us; skew 500 ppm
I20260928 05:25:53.453639 21900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:53 dist-test-slave-515x krb5kdc[20543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573153, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:53.457706 21900 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:25:53.459585 21900 webserver.cc:460] Webserver started at http://127.8.124.195:38157/ using document root <none> and password file <none>
I20260928 05:25:53.460630 21900 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:53.460799 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:25:53 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573153, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:53.499619 21900 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.006s
I20260928 05:25:53.503400 21916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:53.505330 21900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:25:53.505581 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "eef02ead408d471f9bccd6d601439ffc"
format_stamp: "Formatted at 2026-09-28 05:25:50 on dist-test-slave-515x"
server_key: "d82009b3b5e455b23d98f4cda8f54b3f"
server_key_iv: "f5333b6d030c410a88dbcba113e5f073"
server_key_version: "kuduclusterkey@0"
I20260928 05:25:53.507102 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:53.532965 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:53.537884 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:53.538188 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:53.539424 21900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:25:53.541877 21900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:25:53.541987 21900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:53.542094 21900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:25:53.542166 21900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:25:53 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573152, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for HTTP/127.8.124.212@KRBTEST.COM
I20260928 05:25:53.820755 21900 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:33919
I20260928 05:25:53.820817 22031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:33919 every 8 connection(s)
I20260928 05:25:53.822789 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:25:53.830163 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 21900
Sep 28 05:25:53 dist-test-slave-515x krb5kdc[20543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573153, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:53.849908 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.826966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:60245 (local address 127.8.124.254:33045)
0928 05:25:53.831237 (+ 4271us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.831243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.831264 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:25:53.831327 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.831332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.831431 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.831552 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:53.834120 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.835324 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.837818 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.838537 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.838654 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.842934 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:53.842941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:53.842946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:53.842985 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:53.846623 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.847485 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.847490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.847493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.847631 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.848094 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.848097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.848099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.848363 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:53.848489 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.849296 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.849500 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4150,"thread_start_us":4320,"threads_started":1}
I20260928 05:25:53.851159 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.826365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33045 (local address 127.8.124.195:60245)
0928 05:25:53.828281 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:53.828328 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:53.830858 (+ 2530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:53.832419 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:53.832433 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:53.833068 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:53.833902 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.833926 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.835516 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.835522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.837616 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.837632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.839001 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.839008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.839315 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.840079 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:53.840111 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:53.842660 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:53.846758 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:53.846767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:53.846785 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:53.847353 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:53.847736 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:53.847740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:53.847743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:53.847981 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:53.848487 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:53.848494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:53.848863 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 05:25:53.849162 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 05:25:53.849740 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"thread_start_us":1351,"threads_started":1}
I20260928 05:25:53.852712 22032 heartbeater.cc:347] Connected to a master server at 127.8.124.254:33045
I20260928 05:25:53.853027 22032 heartbeater.cc:464] Registering TS with master...
I20260928 05:25:53.853816 22032 heartbeater.cc:511] Master 127.8.124.254:33045 requested a full tablet report, sending...
I20260928 05:25:53.855202 21554 ts_manager.cc:194] Registered new tserver with Master: eef02ead408d471f9bccd6d601439ffc (127.8.124.195:33919)
I20260928 05:25:53.856240 21554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:60245
I20260928 05:25:53.858388 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.831139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.8.124.254:33045)
0928 05:25:53.843070 (+ 11931us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.843075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.843092 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:25:53.843137 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.843142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.843231 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.843330 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:53.845412 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.846290 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.847895 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.848584 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.848700 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.852096 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:53.852104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:53.852109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:53.852145 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:53.855355 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.856134 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.856140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.856144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.856291 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.856703 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.856708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.856711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.857029 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:53.857347 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.857636 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.857849 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11790,"thread_start_us":11041,"threads_started":1}
I20260928 05:25:53.861064 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:25:53.878387 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.864205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.8.124.254:33045)
0928 05:25:53.864427 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.864433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.864721 (+ 288us) server_negotiation.cc:407] Connection header received
0928 05:25:53.864884 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.864887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.864957 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.865050 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:53.867003 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.867698 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.869521 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.870006 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.870119 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.873159 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:53.873164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:53.873167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:53.873198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:53.875745 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.876540 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.876543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.876546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.876659 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:53.877026 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:53.877029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:53.877031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:53.877340 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:53.877519 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.877909 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.878079 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:25:53.883582 21554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:25:53.890888 21554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:25:53.928460 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.904143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32810)
0928 05:25:53.915516 (+ 11373us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:53.915560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:53.915759 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:53.920397 (+ 4638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:53.920405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:53.920437 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:53.920869 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.920881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.923089 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.923095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.925558 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.925578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.927478 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.927484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.927710 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.927767 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 05:25:53.927872 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:25:53.928066 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11095,"threads_started":1}
I20260928 05:25:53.931452 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.904747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:33919 (local address 127.0.0.1:54514)
0928 05:25:53.918536 (+ 13789us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:53.918555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:53.918679 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:53.920863 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:53.920868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:53.920896 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:53.921314 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.921325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.924565 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.924571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.926900 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.926917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.930577 (+ 3660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.930583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.930761 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.930796 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:25:53.930882 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:25:53.931082 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13675,"thread_start_us":11185,"threads_started":1}
I20260928 05:25:53.933895 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.905631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.8.124.195:33919)
0928 05:25:53.917806 (+ 12175us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.917815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.918798 (+ 983us) server_negotiation.cc:407] Connection header received
0928 05:25:53.918908 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.918916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.919197 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.920532 (+ 1335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:53.921518 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.924396 (+ 2878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.928387 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.930010 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.930156 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.930419 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.933311 (+ 2892us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.933516 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12010,"thread_start_us":11845,"threads_started":1}
I20260928 05:25:53.936365 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.904840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:45195 (local address 127.0.0.1:54002)
0928 05:25:53.928933 (+ 24093us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:53.928952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:53.929053 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:53.930044 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:53.930049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:53.930074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:53.930248 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.930258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.932032 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.932037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.933928 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:53.933942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.935633 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.935639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:53.935852 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.935886 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:25:53.935983 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:25:53.936107 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23980,"thread_start_us":13479,"threads_started":1}
I20260928 05:25:53.936590 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.906039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.8.124.194:45195)
0928 05:25:53.918144 (+ 12105us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.918152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.929178 (+ 11026us) server_negotiation.cc:407] Connection header received
0928 05:25:53.929284 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.929293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.929554 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.929824 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:53.930404 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.931878 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.934093 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.935456 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.935588 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.935791 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.936176 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.936334 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11976,"thread_start_us":11891,"threads_started":1}
I20260928 05:25:53.936977 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:53.904309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.8.124.193:33763)
0928 05:25:53.919586 (+ 15277us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:53.919594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:53.919622 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:25:53.919749 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:53.919757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:53.919966 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:53.921103 (+ 1137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:53.921320 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.922945 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.925802 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:53.927232 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:53.936118 (+ 8886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:53.936384 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:53.936512 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 05:25:53.936692 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15111,"thread_start_us":15818,"threads_started":1}
I20260928 05:25:53.938385 21965 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 (DEFAULT_TABLE table=test-table [id=3962de6c75384bf3bfa8983194439b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:53.939980 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec551d4181b494d9f3f7a9ecd781753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:53.939896 21830 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 (DEFAULT_TABLE table=test-table [id=3962de6c75384bf3bfa8983194439b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:53.940326 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 (DEFAULT_TABLE table=test-table [id=3962de6c75384bf3bfa8983194439b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:25:53.941119 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec551d4181b494d9f3f7a9ecd781753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:53.941565 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec551d4181b494d9f3f7a9ecd781753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:53.961625 22058 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Bootstrap starting.
I20260928 05:25:53.964306 22059 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Bootstrap starting.
I20260928 05:25:53.968744 22058 tablet_bootstrap.cc:654] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:53.968744 22059 tablet_bootstrap.cc:654] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:53.970013 22058 log.cc:824] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:53.970297 22060 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Bootstrap starting.
I20260928 05:25:53.973908 22060 tablet_bootstrap.cc:654] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:53.974073 22059 log.cc:824] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:53.975196 22060 log.cc:824] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:53.981123 22058 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: No bootstrap required, opened a new log
I20260928 05:25:53.981582 22058 ts_tablet_manager.cc:1397] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20260928 05:25:53.990940 21896 heartbeater.cc:503] Master 127.8.124.254:33045 was elected leader, sending a full tablet report...
I20260928 05:25:53.994118 22060 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: No bootstrap required, opened a new log
I20260928 05:25:53.994581 22060 ts_tablet_manager.cc:1397] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.013s
I20260928 05:25:53.995397 22058 raft_consensus.cc:348] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:53.996557 22058 raft_consensus.cc:374] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:53.996685 22058 raft_consensus.cc:729] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef02ead408d471f9bccd6d601439ffc, State: Initialized, Role: FOLLOWER
I20260928 05:25:53.997467 22058 consensus_queue.cc:260] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:53.997797 22059 tablet_bootstrap.cc:492] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: No bootstrap required, opened a new log
I20260928 05:25:53.998078 22059 ts_tablet_manager.cc:1397] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.004s
I20260928 05:25:54.005362 22060 raft_consensus.cc:348] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.005932 22060 raft_consensus.cc:374] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:54.006145 22060 raft_consensus.cc:729] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e059157282c4fd7bbfd5beedd673cb1, State: Initialized, Role: FOLLOWER
I20260928 05:25:54.006943 22060 consensus_queue.cc:260] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.007747 22059 raft_consensus.cc:348] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.008210 22059 raft_consensus.cc:374] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:54.008217 22032 heartbeater.cc:503] Master 127.8.124.254:33045 was elected leader, sending a full tablet report...
I20260928 05:25:54.008347 22059 raft_consensus.cc:729] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af08b01ce214a33b377b8965ae115bc, State: Initialized, Role: FOLLOWER
I20260928 05:25:54.008977 22059 consensus_queue.cc:260] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.010926 22058 ts_tablet_manager.cc:1428] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Time spent starting tablet: real 0.029s user 0.010s sys 0.006s
I20260928 05:25:54.011811 22064 raft_consensus.cc:482] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:25:54.012595 22064 raft_consensus.cc:504] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.017385 22059 ts_tablet_manager.cc:1428] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260928 05:25:54.017515 22064 leader_election.cc:288] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5af08b01ce214a33b377b8965ae115bc (127.8.124.193:33763), 6e059157282c4fd7bbfd5beedd673cb1 (127.8.124.194:45195)
I20260928 05:25:54.017777 21751 heartbeater.cc:503] Master 127.8.124.254:33045 was elected leader, sending a full tablet report...
I20260928 05:25:54.022029 22060 ts_tablet_manager.cc:1428] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Time spent starting tablet: real 0.027s user 0.011s sys 0.005s
I20260928 05:25:54.023556 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.016933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:45195 (local address 127.8.124.195:39513)
0928 05:25:54.017107 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.017128 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:54.017246 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:54.018009 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:54.018014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:54.018039 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:54.018226 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:54.018237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.019497 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.019502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:54.021392 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:54.021405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.022890 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.022896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:54.023065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.023097 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:25:54.023177 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:25:54.023308 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:25:54.024161 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.017380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:39513 (local address 127.8.124.194:45195)
0928 05:25:54.017564 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.017570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.017588 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:25:54.017679 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.017683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.017749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.017846 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:54.018381 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.019345 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.021588 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.022743 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.022897 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.022971 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.023810 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.023929 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:25:54.025061 21752 tablet.cc:2369] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:25:54.025480 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec551d4181b494d9f3f7a9ecd781753" candidate_uuid: "eef02ead408d471f9bccd6d601439ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e059157282c4fd7bbfd5beedd673cb1" is_pre_election: true
I20260928 05:25:54.026054 21850 raft_consensus.cc:2454] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef02ead408d471f9bccd6d601439ffc in term 0.
I20260928 05:25:54.026731 21918 leader_election.cc:302] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e059157282c4fd7bbfd5beedd673cb1, eef02ead408d471f9bccd6d601439ffc; no voters:
I20260928 05:25:54.027079 22065 raft_consensus.cc:2781] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:25:54.027302 22065 raft_consensus.cc:482] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:25:54.027526 22065 raft_consensus.cc:3037] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:54.030359 22065 raft_consensus.cc:504] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.031320 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.018453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:57227 (local address 127.8.124.193:33763)
0928 05:25:54.018631 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.018637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.025089 (+ 6452us) server_negotiation.cc:407] Connection header received
0928 05:25:54.025414 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.025420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.025488 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.025545 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:25:54.026087 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.027087 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.029250 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.030457 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.030620 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.030701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.030976 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.031104 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 05:25:54.031423 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.018722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.8.124.195:57227)
0928 05:25:54.024992 (+ 6270us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.025011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:54.025254 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:54.025762 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:54.025767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:54.025791 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:25:54.025949 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:54.025958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.027249 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.027253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:54.029080 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:54.029094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.030646 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.030652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:54.030818 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.030850 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:25:54.030934 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:25:54.031053 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6175,"thread_start_us":6407,"threads_started":1}
I20260928 05:25:54.031522 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec551d4181b494d9f3f7a9ecd781753" candidate_uuid: "eef02ead408d471f9bccd6d601439ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e059157282c4fd7bbfd5beedd673cb1"
I20260928 05:25:54.031718 21850 raft_consensus.cc:3037] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:54.033743 21850 raft_consensus.cc:2454] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eef02ead408d471f9bccd6d601439ffc in term 1.
I20260928 05:25:54.033921 22065 leader_election.cc:288] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [CANDIDATE]: Term 1 election: Requested vote from peers 5af08b01ce214a33b377b8965ae115bc (127.8.124.193:33763), 6e059157282c4fd7bbfd5beedd673cb1 (127.8.124.194:45195)
I20260928 05:25:54.034096 21918 leader_election.cc:302] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e059157282c4fd7bbfd5beedd673cb1, eef02ead408d471f9bccd6d601439ffc; no voters:
I20260928 05:25:54.034446 22065 raft_consensus.cc:2781] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:25:54.034638 22065 raft_consensus.cc:686] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 1 LEADER]: Becoming Leader. State: Replica: eef02ead408d471f9bccd6d601439ffc, State: Running, Role: LEADER
I20260928 05:25:54.034646 21705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec551d4181b494d9f3f7a9ecd781753" candidate_uuid: "eef02ead408d471f9bccd6d601439ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af08b01ce214a33b377b8965ae115bc"
I20260928 05:25:54.034597 21706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec551d4181b494d9f3f7a9ecd781753" candidate_uuid: "eef02ead408d471f9bccd6d601439ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af08b01ce214a33b377b8965ae115bc" is_pre_election: true
I20260928 05:25:54.035122 21706 raft_consensus.cc:2454] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eef02ead408d471f9bccd6d601439ffc in term 0.
I20260928 05:25:54.035157 22065 consensus_queue.cc:237] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } }
I20260928 05:25:54.039933 21554 catalog_manager.cc:5495] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to eef02ead408d471f9bccd6d601439ffc (127.8.124.195). New cstate: current_term: 1 leader_uuid: "eef02ead408d471f9bccd6d601439ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eef02ead408d471f9bccd6d601439ffc" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 } health_report { overall_health: UNKNOWN } } }
I20260928 05:25:54.062634 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.057446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.8.124.195:33919)
0928 05:25:54.057664 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.057669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.057688 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:25:54.057954 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.057958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.058045 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.058166 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:25:54.058532 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.059231 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.060586 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.060966 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.061069 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.061539 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:25:54.062083 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:25:54.062224 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.062318 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.062395 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:25:54.070318 21705 raft_consensus.cc:3037] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:54.070694 21850 raft_consensus.cc:1261] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 1 FOLLOWER]: Refusing update from remote peer eef02ead408d471f9bccd6d601439ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:54.071659 22064 consensus_queue.cc:1035] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e059157282c4fd7bbfd5beedd673cb1" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:25:54.073333 21705 raft_consensus.cc:1261] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 1 FOLLOWER]: Refusing update from remote peer eef02ead408d471f9bccd6d601439ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:54.074155 22064 consensus_queue.cc:1035] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af08b01ce214a33b377b8965ae115bc" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 33763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:25:54.081019 22033 tablet.cc:2369] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:25:54.091897 22071 mvcc.cc:205] Tried to move back new op lower bound from 7334187639063244800 to 7334187638932631552. Current Snapshot: MvccSnapshot[applied={T|T < 7334187639063244800}]
I20260928 05:25:54.096585 22074 mvcc.cc:205] Tried to move back new op lower bound from 7334187639063244800 to 7334187638932631552. Current Snapshot: MvccSnapshot[applied={T|T < 7334187639063244800}]
I20260928 05:25:54.098693 22072 mvcc.cc:205] Tried to move back new op lower bound from 7334187639063244800 to 7334187638932631552. Current Snapshot: MvccSnapshot[applied={T|T < 7334187639063244800}]
I20260928 05:25:54.112025 21553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:25:54.112388 21553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:25:54.116761 21553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f4466e71274415854a2bedc911ccd8: Sending DeleteTablet for 3 replicas of tablet 8ec551d4181b494d9f3f7a9ecd781753
I20260928 05:25:54.117919 21686 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:54 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:32810
I20260928 05:25:54.118700 21830 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:54 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:54002
I20260928 05:25:54.118796 21965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec551d4181b494d9f3f7a9ecd781753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:25:54 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:54514
I20260928 05:25:54.126113 22081 tablet_replica.cc:330] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: stopping tablet replica
I20260928 05:25:54.126439 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21591
I20260928 05:25:54.126611 22081 raft_consensus.cc:2229] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:25:54.129937 22081 raft_consensus.cc:2258] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:54.138880 22081 ts_tablet_manager.cc:1905] T 8ec551d4181b494d9f3f7a9ecd781753 P 5af08b01ce214a33b377b8965ae115bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:54.139663 22083 tablet_replica.cc:330] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: stopping tablet replica
I20260928 05:25:54.140694 22083 raft_consensus.cc:2229] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:25:54.142035 22083 raft_consensus.cc:2258] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:54.143105 22082 tablet_replica.cc:330] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: stopping tablet replica
I20260928 05:25:54.143843 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.127586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.8.124.193:33763)
0928 05:25:54.127744 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.127749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.127780 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:25:54.127838 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.127843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.127912 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.128020 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:54.129835 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.131084 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.133065 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.133701 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.133831 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.137131 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:54.137144 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:54.137163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:54.137207 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:54.140909 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.141745 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.141754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.141767 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.141924 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.142321 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.142327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.142330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.142729 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:54.143040 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.143282 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.143487 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:25:54.143899 22082 raft_consensus.cc:2229] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:25:54.144927 21746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:54.146545 22083 ts_tablet_manager.cc:1905] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:54.147092 22082 raft_consensus.cc:2258] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:25:54.152160 22082 ts_tablet_manager.cc:1905] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:25:54.159519 22082 ts_tablet_manager.cc:1918] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:25:54.164187 22083 ts_tablet_manager.cc:1918] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:25:54.165115 22083 log.cc:1196] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/8ec551d4181b494d9f3f7a9ecd781753
I20260928 05:25:54.166272 22082 log.cc:1196] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/8ec551d4181b494d9f3f7a9ecd781753
I20260928 05:25:54.166440 22083 ts_tablet_manager.cc:1939] T 8ec551d4181b494d9f3f7a9ecd781753 P eef02ead408d471f9bccd6d601439ffc: Deleting consensus metadata
I20260928 05:25:54.166815 22082 ts_tablet_manager.cc:1939] T 8ec551d4181b494d9f3f7a9ecd781753 P 6e059157282c4fd7bbfd5beedd673cb1: Deleting consensus metadata
I20260928 05:25:54.168488 21539 catalog_manager.cc:4841] TS 6e059157282c4fd7bbfd5beedd673cb1 (127.8.124.194:45195): tablet 8ec551d4181b494d9f3f7a9ecd781753 (table test-table [id=3962de6c75384bf3bfa8983194439b13]) successfully deleted
I20260928 05:25:54.168440 21538 catalog_manager.cc:4841] TS eef02ead408d471f9bccd6d601439ffc (127.8.124.195:33919): tablet 8ec551d4181b494d9f3f7a9ecd781753 (table test-table [id=3962de6c75384bf3bfa8983194439b13]) successfully deleted
I20260928 05:25:54.267591 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21591
W20260928 05:25:54.280691 21541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:33763 (error 108)
I20260928 05:25:54.281350 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21755
I20260928 05:25:54.282809 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.282229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32826)
0928 05:25:54.282491 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.282612 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260928 05:25:54.283088 21541 catalog_manager.cc:4584] TS 5af08b01ce214a33b377b8965ae115bc (127.8.124.193:33763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ec551d4181b494d9f3f7a9ecd781753: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
I20260928 05:25:54.298954 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.282153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.8.124.194:45195)
0928 05:25:54.282333 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.282339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.282518 (+ 179us) server_negotiation.cc:407] Connection header received
0928 05:25:54.282707 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.282713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.282782 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.282861 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:54.284284 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.285008 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.286513 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.286909 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.287016 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.293403 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:54.293417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:54.293432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:54.293467 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:54.296372 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.296994 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.297001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.297003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.297124 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.297474 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.297479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.297481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.297863 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:54.298121 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.298416 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.298646 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:25:54.299559 21891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:54.336712 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.336184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32842)
0928 05:25:54.336414 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.336534 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 05:25:54.337913 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.337447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32846)
0928 05:25:54.337633 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.337732 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:25:54.386778 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21755
I20260928 05:25:54.398377 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.397954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32854)
0928 05:25:54.398150 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.398235 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:25:54.399320 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.399001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32868)
0928 05:25:54.399140 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.399208 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 05:25:54.406296 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21900
I20260928 05:25:54.421146 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.407040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.8.124.195:33919)
0928 05:25:54.407213 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:54.407219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:54.407239 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:25:54.407388 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:54.407392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:54.407477 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:54.407557 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:54.408863 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.409667 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.410989 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:54.411686 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:54.411848 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:54.414468 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:54.414481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:54.414501 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:54.414546 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:54.418332 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.418982 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.418991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.418995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.419187 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:54.419585 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:54.419590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:54.419593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:54.420015 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:54.420339 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:54.420560 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 05:25:54.420779 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:25:54.421870 22025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:54.479843 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.479334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32870)
0928 05:25:54.479566 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.479679 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 05:25:54.480813 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.480447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32872)
0928 05:25:54.480610 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.480675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 05:25:54.538745 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21900
I20260928 05:25:54.553550 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 21521
I20260928 05:25:54.554662 21584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:54.663684 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:54.663224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:33763 (local address 127.0.0.1:32884)
0928 05:25:54.663383 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:54.663479 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:33763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 05:25:54.663686 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 21521
I20260928 05:25:54.680925 8691 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:25:55.014618 8691 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:25:55.014837 8691 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:25:55.349061 8691 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:25:55.349 UTC [20573] LOG: received smart shutdown request
2026-09-28 05:25:55.352 UTC [20573] LOG: background worker "logical replication launcher" (PID 20582) exited with exit code 1
2026-09-28 05:25:55.352 UTC [20577] LOG: shutting down
2026-09-28 05:25:55.586 UTC [20573] LOG: database system is shut down
2026-09-28T05:25:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50961 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:25:55 dist-test-slave-515x krb5kdc[22097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:25:55 dist-test-slave-515x krb5kdc[22097](info): set up 2 sockets
Sep 28 05:25:55 dist-test-slave-515x krb5kdc[22097](info): commencing operation
krb5kdc: starting...
W20260928 05:25:56.689625 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:25:56 dist-test-slave-515x krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:25:56.894582 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38713
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:25:57.212317 22110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:57.212604 22110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:57.212744 22110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:57.220093 22110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:25:57.220208 22110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:57.220283 22110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:57.220362 22110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:25:57.220419 22110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:25:57.230388 22110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:57.231714 22110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:57.232993 22110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:57.244035 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:57.243999 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:57.243973 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:57.245069 22110 server_base.cc:1034] running on GCE node
I20260928 05:25:57.245870 22110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:57.247283 22110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:57.248471 22110 hybrid_clock.cc:648] HybridClock initialized: now 1790573157248432 us; error 32 us; skew 500 ppm
I20260928 05:25:57.248802 22110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:57 dist-test-slave-515x krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:57.251847 22110 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:25:57.253306 22110 webserver.cc:460] Webserver started at http://127.8.124.254:43797/ using document root <none> and password file <none>
I20260928 05:25:57.253997 22110 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:57.254101 22110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:57.254385 22110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:25:57.257218 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "05849924f9864f2d92f858a1b73627ea"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "677bd8a4c920334233eb76b1ee01a467"
server_key_iv: "5eaf13b7d183ebc9d9252963eeea3092"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.258011 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "05849924f9864f2d92f858a1b73627ea"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "677bd8a4c920334233eb76b1ee01a467"
server_key_iv: "5eaf13b7d183ebc9d9252963eeea3092"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.262962 22110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:25:57.266361 22125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:57.267839 22110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:25:57.267997 22110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "05849924f9864f2d92f858a1b73627ea"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "677bd8a4c920334233eb76b1ee01a467"
server_key_iv: "5eaf13b7d183ebc9d9252963eeea3092"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.268191 22110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:57.304068 22110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:57.308339 22110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:57.308596 22110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:57.325857 22110 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:38713
I20260928 05:25:57.325876 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:38713 every 8 connection(s)
I20260928 05:25:57.327271 22110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:25:57.330938 22177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:57.333503 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22110
I20260928 05:25:57.333762 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:25:57.334538 8691 external_mini_cluster.cc:1428] Setting key 4d51f28ee30a196819c15c9bc42b8e4d
I20260928 05:25:57.339129 22177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:25:57 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.253@KRBTEST.COM" created.
I20260928 05:25:57.361639 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.343199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38713 (local address 127.0.0.1:52956)
0928 05:25:57.344170 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.344226 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:57.346005 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:57.346800 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:57.346806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:57.347471 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:57.347813 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.347831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.349310 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.349314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.350421 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.350433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.351395 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.351402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.351636 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.352214 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:57.352236 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:57.354518 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:57.357903 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.357913 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.357923 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.358289 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.358725 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.358728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.358731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.358871 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.359752 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:57.359758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:57.359950 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 05:25:57.360208 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 05:25:57.360581 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":52,"thread_start_us":383,"threads_started":1}
I20260928 05:25:57.361661 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.343199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.8.124.254:38713)
0928 05:25:57.344228 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:57.344239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:57.344333 (+ 94us) server_negotiation.cc:407] Connection header received
0928 05:25:57.346350 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:57.346356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:57.346581 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:57.346775 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:57.347995 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.349175 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.350550 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.351179 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.351341 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.354714 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:57.354722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:57.354733 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:57.354775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:57.357774 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.358428 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.358436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.358439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.358621 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.358961 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.358965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.358968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.359571 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:57.359761 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:57.360357 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 05:25:57.360723 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":836,"spinlock_wait_cycles":59264,"thread_start_us":613,"threads_started":1}
I20260928 05:25:57.364539 22177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:25:57.365584 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.365235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60366)
0928 05:25:57.365384 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.365482 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:25:57.365867 22129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
I20260928 05:25:57.367283 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.366972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60380)
0928 05:25:57.367100 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.367167 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:25:57.367450 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:38695: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
I20260928 05:25:57.382591 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:38695
--webserver_interface=127.8.124.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:25:57.416738 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } attempt: 1
I20260928 05:25:57.417889 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.417432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60396)
0928 05:25:57.417629 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.417740 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 05:25:57.418785 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.418449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60402)
0928 05:25:57.418583 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.418649 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:25:57.418944 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:38695: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
I20260928 05:25:57.487174 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } attempt: 2
I20260928 05:25:57.488250 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.487785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60404)
0928 05:25:57.488012 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.488100 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:25:57.489131 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.488808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60418)
0928 05:25:57.488937 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.489012 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:25:57.489295 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:38695: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
I20260928 05:25:57.580518 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } attempt: 3
I20260928 05:25:57.581521 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.581104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60432)
0928 05:25:57.581293 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.581395 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 05:25:57.582463 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.582122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60436)
0928 05:25:57.582261 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.582332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:25:57.582624 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:38695: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
W20260928 05:25:57.690318 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:57.690608 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:57.690742 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:57.697831 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:25:57.697952 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:57.698028 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:57.698101 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:25:57.698165 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:25:57.708050 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.124.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:57.709326 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:57.710625 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:57.721290 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:57.721354 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:57.721330 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:57.721897 22184 server_base.cc:1034] running on GCE node
I20260928 05:25:57.722827 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:57.724025 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:57.725227 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790573157725171 us; error 47 us; skew 500 ppm
I20260928 05:25:57.725541 22184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:25:57.725823 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } attempt: 4
I20260928 05:25:57.726869 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.726352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60438)
0928 05:25:57.726660 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.726739 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 05:25:57.727829 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.727417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60444)
0928 05:25:57.727569 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.727685 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:25:57.728039 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:38695: Network error: Client connection negotiation failed: client connection to 127.8.124.253:38695: connect: Connection refused (error 111)
Sep 28 05:25:57 dist-test-slave-515x krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:57.729292 22184 init.cc:377] Logged in from keytab as kudu/127.8.124.253@KRBTEST.COM (short username kudu)
I20260928 05:25:57.730651 22184 webserver.cc:460] Webserver started at http://127.8.124.253:45939/ using document root <none> and password file <none>
I20260928 05:25:57.731251 22184 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:57.731344 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:57.731570 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:25:57.734385 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data/instance:
uuid: "2ae94b6518c941a0b4b5104104b2d96e"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "6a67b051bf20c6967d645b8b4ab95bd3"
server_key_iv: "7c468c9dd978b37939f608ef7b9149cc"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.735085 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal/instance:
uuid: "2ae94b6518c941a0b4b5104104b2d96e"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "6a67b051bf20c6967d645b8b4ab95bd3"
server_key_iv: "7c468c9dd978b37939f608ef7b9149cc"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.739787 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:25:57.743016 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:57.744292 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:25:57.744443 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "2ae94b6518c941a0b4b5104104b2d96e"
format_stamp: "Formatted at 2026-09-28 05:25:57 on dist-test-slave-515x"
server_key: "6a67b051bf20c6967d645b8b4ab95bd3"
server_key_iv: "7c468c9dd978b37939f608ef7b9149cc"
server_key_version: "encryptionkey@0"
I20260928 05:25:57.744616 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:57.763187 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:57.767128 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:57.767387 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:57.784042 22184 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.253:38695
I20260928 05:25:57.784065 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.253:38695 every 8 connection(s)
I20260928 05:25:57.785368 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
I20260928 05:25:57.788897 22251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:57.789809 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22184
I20260928 05:25:57.789969 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-1/wal/instance
I20260928 05:25:57.790397 8691 external_mini_cluster.cc:1428] Setting key 404d9a7b950aecbc574e71a1609371f9
I20260928 05:25:57.796352 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.124.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.252@KRBTEST.COM" created.
Sep 28 05:25:57 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:57.821434 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.798829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.8.124.254:38713)
0928 05:25:57.799028 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:57.799033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:57.800190 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 05:25:57.802007 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:57.802012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:57.802088 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:57.802190 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:57.803552 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.804541 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.806037 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.806661 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.806797 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.812185 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:57.812190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:57.812194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:57.812226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:57.818448 (+ 6222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.819307 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.819310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.819312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.819422 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.819817 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.819819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.819821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.820084 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:57.820211 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:57.821034 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 05:25:57.821207 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:25:57.822882 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.799251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38713 (local address 127.0.0.1:52960)
0928 05:25:57.800029 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.800072 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:57.801770 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:57.802371 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:57.802378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:57.802872 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:57.803402 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.803418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.804662 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.804666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.805902 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.805914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.806856 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.806861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.807033 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.807659 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:57.807680 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:57.811968 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:57.818609 (+ 6641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.818619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.818635 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.819208 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.819507 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.819512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.819515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.819723 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.820186 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:57.820194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:57.820627 (+ 433us) client_negotiation.cc:769] Sending connection context
0928 05:25:57.820927 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 05:25:57.821440 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":314,"threads_started":1}
I20260928 05:25:57.824554 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:25:57 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
I20260928 05:25:57.837492 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:34761
--webserver_interface=127.8.124.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:25:57.838569 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.825357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60448)
0928 05:25:57.825539 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.825559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:57.825707 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:57.826572 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:57.826575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:57.826938 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:57.827076 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.827084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.828965 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.828969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.829859 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:57.829870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.830674 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.830679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:57.830844 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.831432 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:57.831448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:57.833341 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:57.836226 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.836232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.836235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.836675 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.837074 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:57.837078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:57.837080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:57.837204 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:57.837905 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:57.837908 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:57.838012 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 05:25:57.838117 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:25:57.838319 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:25:57.838920 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.825406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.8.124.253:38695)
0928 05:25:57.826044 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:57.826053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:57.826084 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:25:57.826139 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:57.826148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:57.826398 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:57.826582 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:57.827219 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.828839 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.830000 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:57.830553 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:57.830645 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:57.833548 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:57.833559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:57.833572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:57.833620 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:57.836124 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.836801 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.836810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.836813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.836962 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:57.837303 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:57.837308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:57.837311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:57.837781 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:57.837971 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:57.838267 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 05:25:57.838510 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":287,"threads_started":1}
I20260928 05:25:57.841269 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:25:57.842289 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.841840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36722)
0928 05:25:57.841996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.842120 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:25:57.842514 22203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:57.844069 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.843695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36726)
0928 05:25:57.843855 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.843971 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:25:57.844265 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:57.893558 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 1
I20260928 05:25:57.894618 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.894184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36734)
0928 05:25:57.894365 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.894467 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 05:25:57.895566 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.895282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36750)
0928 05:25:57.895393 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.895468 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:25:57.895773 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:57.963989 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 2
I20260928 05:25:57.965029 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.964605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36762)
0928 05:25:57.964800 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.964884 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:25:57.966044 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:57.965745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36772)
0928 05:25:57.965870 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:57.965937 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:25:57.966230 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:58.027297 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } attempt: 5
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:25:58.040287 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.027987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60458)
0928 05:25:58.028194 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.028215 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.028361 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.028724 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.028730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.029102 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.029280 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.029291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.030332 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.030336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.031171 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.031183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.031876 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.031881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.032055 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.032605 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.032622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.034592 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.038083 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.038087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.038090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.038427 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.038787 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.038790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.038792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.038908 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.039531 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.039535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.039702 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.039832 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.040020 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:25:58.040516 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.028129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.8.124.253:38695)
0928 05:25:58.028299 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.028305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.028327 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:25:58.028507 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.028511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.028588 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.028667 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.029436 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.030197 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.031314 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.031734 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.031849 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.034786 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.034793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.034798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.034840 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.037968 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.038530 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.038536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.038540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.038681 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.038995 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.038999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.039003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.039381 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.039561 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.039966 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.040202 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:25:58.041564 22177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:25:58.042548 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.042194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36776)
0928 05:25:58.042353 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.042424 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:25:58.043401 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.043103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36790)
0928 05:25:58.043231 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.043311 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:25:58.043594 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:58.057413 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 3
I20260928 05:25:58.058444 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.058010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36798)
0928 05:25:58.058208 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.058308 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:25:58.059448 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.059057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36814)
0928 05:25:58.059272 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.059343 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260928 05:25:58.059650 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:58.094806 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 1
I20260928 05:25:58.095958 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.095506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36820)
0928 05:25:58.095704 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.095803 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:25:58.096966 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.096608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36830)
0928 05:25:58.096771 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.096843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:25:58.097213 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
W20260928 05:25:58.149214 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:25:58.149492 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:25:58.149636 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:25:58.156576 22258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:25:58.156682 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:25:58.156765 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:25:58.156836 22258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:25:58.156895 22258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:25:58.165407 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 2
I20260928 05:25:58.166369 22258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38713,127.8.124.253:38695,127.8.124.252:34761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.124.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:25:58.166462 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.166013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36834)
0928 05:25:58.166215 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.166306 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:25:58.167515 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.167205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36846)
0928 05:25:58.167348 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.167414 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:25:58.167692 22177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:58.167778 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:25:58.169004 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:25:58.179921 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:58.179984 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:25:58.179958 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:25:58.180949 22258 server_base.cc:1034] running on GCE node
I20260928 05:25:58.181682 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:25:58.182986 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:25:58.184180 22258 hybrid_clock.cc:648] HybridClock initialized: now 1790573158184130 us; error 42 us; skew 500 ppm
I20260928 05:25:58.184504 22258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573158, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:25:58.187304 22258 init.cc:377] Logged in from keytab as kudu/127.8.124.252@KRBTEST.COM (short username kudu)
I20260928 05:25:58.188654 22258 webserver.cc:460] Webserver started at http://127.8.124.252:41415/ using document root <none> and password file <none>
I20260928 05:25:58.189322 22258 fs_manager.cc:359] Metadata directory not provided
I20260928 05:25:58.189435 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:25:58.189744 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:25:58.192555 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data/instance:
uuid: "f4de5a5ad7f346efb8f7ab26f411d710"
format_stamp: "Formatted at 2026-09-28 05:25:58 on dist-test-slave-515x"
server_key: "07caab20e1cb3087342cf61d1119b9e5"
server_key_iv: "f7c1dda8fbe0fae88412fcbca88b9756"
server_key_version: "encryptionkey@0"
I20260928 05:25:58.193317 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal/instance:
uuid: "f4de5a5ad7f346efb8f7ab26f411d710"
format_stamp: "Formatted at 2026-09-28 05:25:58 on dist-test-slave-515x"
server_key: "07caab20e1cb3087342cf61d1119b9e5"
server_key_iv: "f7c1dda8fbe0fae88412fcbca88b9756"
server_key_version: "encryptionkey@0"
I20260928 05:25:58.198213 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:25:58.201495 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:25:58.202816 22258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:25:58.202826 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 4
I20260928 05:25:58.202976 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "f4de5a5ad7f346efb8f7ab26f411d710"
format_stamp: "Formatted at 2026-09-28 05:25:58 on dist-test-slave-515x"
server_key: "07caab20e1cb3087342cf61d1119b9e5"
server_key_iv: "f7c1dda8fbe0fae88412fcbca88b9756"
server_key_version: "encryptionkey@0"
I20260928 05:25:58.203158 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:25:58.203873 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.203379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36848)
0928 05:25:58.203598 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.203699 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:25:58.204783 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.204443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36854)
0928 05:25:58.204584 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.204665 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 05:25:58.205004 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:34761: Network error: Client connection negotiation failed: client connection to 127.8.124.252:34761: connect: Connection refused (error 111)
I20260928 05:25:58.223091 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:25:58.226744 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:25:58.226980 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:25:58.243202 22258 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.252:34761
I20260928 05:25:58.243264 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.252:34761 every 8 connection(s)
I20260928 05:25:58.244455 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
I20260928 05:25:58.247496 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22258
I20260928 05:25:58.247695 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572965871275-8691-0/minicluster-data/master-2/wal/instance
I20260928 05:25:58.248284 8691 external_mini_cluster.cc:1428] Setting key 2de0810acbe11aad1e06dc373b3393cf
I20260928 05:25:58.248668 22325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:25:58.257707 22325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573158, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:58.271142 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.250049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.8.124.252:34761)
0928 05:25:58.250760 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.250771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.250810 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:25:58.253027 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.253048 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.253438 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.253784 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.255462 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.256708 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.258531 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.259213 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.259355 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.263827 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.263834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.263844 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.263881 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.266974 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.267805 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.267814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.267822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.268008 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.268403 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.268411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.268413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.268955 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.269213 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.269719 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.270123 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":320,"threads_started":1}
I20260928 05:25:58.273887 22177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 3
I20260928 05:25:58.274282 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.260445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38713 (local address 127.0.0.1:52974)
0928 05:25:58.261150 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.261195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.261354 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.261730 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.261738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.262466 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.262735 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.262744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.263715 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.263739 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.264845 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.264858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.265809 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.265814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.265935 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.266489 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.266506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.268860 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.271835 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.271846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.271853 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.272225 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.272580 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.272585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.272589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.272708 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.273206 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.273214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.273389 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.273575 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.273885 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":355,"threads_started":1}
I20260928 05:25:58.274286 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.260620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.8.124.254:38713)
0928 05:25:58.260803 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.260807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.261299 (+ 492us) server_negotiation.cc:407] Connection header received
0928 05:25:58.261460 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.261464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.261526 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.261617 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.262875 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.263563 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.264977 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.265620 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.265749 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.269042 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.269047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.269050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.269084 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.271708 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.272339 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.272344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.272348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.272475 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.272807 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.272812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.272815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.273079 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.273221 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.273718 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.273927 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:25:58.276013 22325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573158, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:25:58.288753 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.274382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36872)
0928 05:25:58.274692 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.274713 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.274817 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.275251 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.275255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.275752 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.275956 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.275967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.277302 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.277308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.278643 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.278657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.279366 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.279372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.279537 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.280161 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.280182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.282803 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.286202 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.286207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.286212 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.286675 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.287048 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.287054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.287058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.287181 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.287840 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.287845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.288038 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.288142 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.288402 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 05:25:58.289849 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.274764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.8.124.252:34761)
0928 05:25:58.274952 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.274957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.274975 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:25:58.275017 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.275022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.275090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.276070 (+ 980us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.276212 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.277191 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.278767 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.279196 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.279347 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.282981 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.282987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.282993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.283029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.286092 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.286812 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.286818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.286822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.286938 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.287313 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.287318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.287321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.287674 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.289124 (+ 1450us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.289215 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.289446 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:25:58.290035 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.276698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:38695 (local address 127.0.0.1:60464)
0928 05:25:58.276913 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.276929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.277008 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.277446 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.277451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.277889 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.278042 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.278051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.279182 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.279187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.280296 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.280308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.281069 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.281075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.281260 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.281780 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.281802 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.283949 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.287269 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.287274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.287277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.287779 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.288236 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.288240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.288243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.288401 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.289031 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.289036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.289209 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.289365 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.289649 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 05:25:58.290059 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.276789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.8.124.253:38695)
0928 05:25:58.276963 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.276968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.276985 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:25:58.277109 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.277114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.277194 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.277286 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.278203 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.279053 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.280429 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.280917 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.281036 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.284122 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.284127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.284130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.284168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.287133 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.287916 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.287921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.287924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.288104 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.288511 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.288515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.288519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.288824 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.288969 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.289453 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.289677 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:25:58.291306 22325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:25:58.292337 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.274969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.8.124.254:38713)
0928 05:25:58.275119 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.275125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.275141 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:25:58.275187 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.275191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.275257 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.275337 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.276909 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.278056 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.280697 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.281306 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.281413 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.285941 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.285946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.285951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.285988 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.289166 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.290130 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.290135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.290138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.290298 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.290756 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.290761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.290765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.291083 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.291348 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.291967 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.292105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 05:25:58.295665 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: Bootstrap starting.
I20260928 05:25:58.299665 22177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:58.300839 22177 log.cc:824] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:25:58.303684 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: No bootstrap required, opened a new log
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573158, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:25:58.308414 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.293884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.8.124.253:38695)
0928 05:25:58.294037 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.294042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.294059 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:25:58.294167 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.294172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.294234 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.294323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.295396 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.296389 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.297708 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.298284 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.298398 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.303102 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.303108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.303113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.303147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.305527 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.306304 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.306309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.306313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.306467 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.306880 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.306886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.306889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.307225 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.307513 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.307873 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.308087 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:25:58.312723 22177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.313122 22177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:58.313251 22177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05849924f9864f2d92f858a1b73627ea, State: Initialized, Role: FOLLOWER
I20260928 05:25:58.313333 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:25:58.313578 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.298061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.8.124.252:34761)
0928 05:25:58.298209 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.298214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.298228 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:25:58.298259 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.298262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.298308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.298380 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.299280 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.300292 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.301909 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.302381 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.302496 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.307909 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.307915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.307919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.307951 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.310866 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.311587 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.311592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.311596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.311744 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.312110 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.312115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.312117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.312431 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.312562 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.312966 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.313253 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":43}
I20260928 05:25:58.313831 22177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.314023 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.297886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36884)
0928 05:25:58.298021 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.298035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.298112 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.298515 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.298520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.298982 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.299136 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.299147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.300438 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.300444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.301753 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.301770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.302523 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.302528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.302718 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.303681 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.303706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.307732 (+ 4026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.310977 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.310982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.310985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.311459 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.311855 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.311860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.311864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.312011 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.312534 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.312539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.312671 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.312832 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.313726 (+ 894us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:25:58.314651 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.315027 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.315749 22177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:25:58.321535 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: Bootstrap starting.
I20260928 05:25:58.336735 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.319475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.8.124.254:38713)
0928 05:25:58.319625 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.319630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.320156 (+ 526us) server_negotiation.cc:407] Connection header received
0928 05:25:58.320560 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.320566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.320637 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.320718 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.322926 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.323966 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.325901 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.326590 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.326688 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.330117 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.330124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.330129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.330168 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.333579 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.334400 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.334405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.334409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.334588 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.335020 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.335026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.335030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.335349 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.335588 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.336129 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.336351 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:25:58.340945 22325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:58.342461 22325 log.cc:824] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:58.345968 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: No bootstrap required, opened a new log
I20260928 05:25:58.356792 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.320297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.8.124.253:38695)
0928 05:25:58.320458 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.320463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.339292 (+ 18829us) server_negotiation.cc:407] Connection header received
0928 05:25:58.339517 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.339523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.339596 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.339664 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.341833 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.342851 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.345465 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.346107 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.346197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.349758 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.349764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.349769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.349806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.353169 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.354295 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.354300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.354304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.354688 (+ 384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.355140 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.355145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.355148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.355494 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.355729 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.356228 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.356446 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:25:58.358011 22325 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.358567 22325 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:58.358733 22325 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de5a5ad7f346efb8f7ab26f411d710, State: Initialized, Role: FOLLOWER
I20260928 05:25:58.359179 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.342161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.8.124.252:34761)
0928 05:25:58.342360 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.342367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.343184 (+ 817us) server_negotiation.cc:407] Connection header received
0928 05:25:58.343406 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.343411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.343487 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.343581 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.345841 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.346870 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.348888 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.349644 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.349736 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.352646 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.352652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.352657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.352697 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.356131 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.356901 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.356906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.356910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.357032 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.357510 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.357515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.357518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.357890 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.358108 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.358641 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.358843 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 05:25:58.359503 22325 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.362118 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.362414 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.363529 22325 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:25:58.378834 22361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:25:58.379038 22361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:25:58.374987 22110 master_runner.cc:186] Error getting master registration for 127.8.124.253:38695: OK, instance_id { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" instance_seqno: 1790573157779244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:25:58.387876 22110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:25:58.390499 22358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:25:58.390691 22358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:25:58.382153 22258 master_runner.cc:186] Error getting master registration for 127.8.124.253:38695: OK, instance_id { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" instance_seqno: 1790573157779244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:25:58.395371 22258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:25:58.478909 22279 tablet.cc:2369] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:25:58.480000 22344 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:25:58.480302 22344 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.481961 22344 leader_election.cc:288] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05849924f9864f2d92f858a1b73627ea (127.8.124.254:38713), 2ae94b6518c941a0b4b5104104b2d96e (127.8.124.253:38695)
I20260928 05:25:58.482717 22152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05849924f9864f2d92f858a1b73627ea" is_pre_election: true
I20260928 05:25:58.483367 22152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4de5a5ad7f346efb8f7ab26f411d710 in term 0.
I20260928 05:25:58.483964 22274 leader_election.cc:302] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05849924f9864f2d92f858a1b73627ea, f4de5a5ad7f346efb8f7ab26f411d710; no voters:
I20260928 05:25:58.484423 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:25:58.484582 22344 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:25:58.484678 22344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:58.488111 22344 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.489341 22152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05849924f9864f2d92f858a1b73627ea"
I20260928 05:25:58.489450 22344 leader_election.cc:288] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [CANDIDATE]: Term 1 election: Requested vote from peers 05849924f9864f2d92f858a1b73627ea (127.8.124.254:38713), 2ae94b6518c941a0b4b5104104b2d96e (127.8.124.253:38695)
I20260928 05:25:58.489566 22152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:58.492993 22152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de5a5ad7f346efb8f7ab26f411d710 in term 1.
I20260928 05:25:58.493517 22274 leader_election.cc:302] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05849924f9864f2d92f858a1b73627ea, f4de5a5ad7f346efb8f7ab26f411d710; no voters:
I20260928 05:25:58.495539 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:25:58.497968 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [term 1 LEADER]: Becoming Leader. State: Replica: f4de5a5ad7f346efb8f7ab26f411d710, State: Running, Role: LEADER
I20260928 05:25:58.498605 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.504159 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } attempt: 5
I20260928 05:25:58.512491 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4de5a5ad7f346efb8f7ab26f411d710. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.513347 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: This master's current role is: LEADER
Sep 28 05:25:58 dist-test-slave-515x krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573157, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:25:58.522073 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.504702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:34761 (local address 127.0.0.1:36908)
0928 05:25:58.505034 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 05:25:58.505054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:25:58.505161 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:25:58.506105 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:25:58.506110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:25:58.506676 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:25:58.506883 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.506896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.508527 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.508533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.509948 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:25:58.509963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.511162 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.511169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:25:58.511376 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.512027 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:25:58.512051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:25:58.514950 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:25:58.519302 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.519308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.519312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.519805 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.520210 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:25:58.520215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:25:58.520219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:25:58.520367 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:25:58.521076 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:25:58.521081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:25:58.521226 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 05:25:58.521373 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 05:25:58.521684 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 05:25:58.523514 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:25:58.505468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.8.124.252:34761)
0928 05:25:58.505662 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:25:58.505668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:25:58.505685 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:25:58.505741 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:25:58.505747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:25:58.505820 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:25:58.505916 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:25:58.507085 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.508276 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.510093 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:25:58.510817 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:25:58.510939 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:25:58.515165 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:25:58.515171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:25:58.515176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:25:58.515214 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:25:58.519150 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.519938 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.519944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.519947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.520085 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:25:58.520487 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:25:58.520492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:25:58.520495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:25:58.520848 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:25:58.521016 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:25:58.522577 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0928 05:25:58.523118 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:25:58.530961 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: Bootstrap starting.
I20260928 05:25:58.534964 22251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: Neither blocks nor log segments found. Creating new log.
I20260928 05:25:58.536244 22251 log.cc:824] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:25:58.536975 22366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:25:58.539130 22366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:25:58.545508 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: No bootstrap required, opened a new log
I20260928 05:25:58.553860 22152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [term 1 FOLLOWER]: Refusing update from remote peer f4de5a5ad7f346efb8f7ab26f411d710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:58.554953 22364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:25:58.559041 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
I20260928 05:25:58.559451 22251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:25:58.559573 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae94b6518c941a0b4b5104104b2d96e, State: Initialized, Role: FOLLOWER
I20260928 05:25:58.560299 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } }
W20260928 05:25:58.561365 22131 tablet.cc:2369] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:25:58.567474 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.567839 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.569473 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:25:58.569540 22225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae94b6518c941a0b4b5104104b2d96e"
I20260928 05:25:58.570045 22225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:25:58.578959 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4de5a5ad7f346efb8f7ab26f411d710. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.579236 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.581316 22226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae94b6518c941a0b4b5104104b2d96e" is_pre_election: true
I20260928 05:25:58.590200 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.590453 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.573531 22225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de5a5ad7f346efb8f7ab26f411d710 in term 1.
I20260928 05:25:58.591325 22224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [term 1 FOLLOWER]: Refusing update from remote peer f4de5a5ad7f346efb8f7ab26f411d710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:25:58.592995 22344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:25:58.606061 22369 mvcc.cc:205] Tried to move back new op lower bound from 7334187657410478080 to 7334187657215275008. Current Snapshot: MvccSnapshot[applied={T|T < 7334187657410478080 or (T in {7334187657410478080})}]
I20260928 05:25:58.614343 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.614693 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:58.616067 22366 catalog_manager.cc:1347] Generated new cluster ID: 8b8be53186834c0aba7c336f5b8c5a61
I20260928 05:25:58.616191 22366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:25:58.623698 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4de5a5ad7f346efb8f7ab26f411d710. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.624004 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.624470 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.624825 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:25:58.655848 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05849924f9864f2d92f858a1b73627ea" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38713 } } peers { permanent_uuid: "2ae94b6518c941a0b4b5104104b2d96e" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 38695 } } peers { permanent_uuid: "f4de5a5ad7f346efb8f7ab26f411d710" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 34761 } } }
I20260928 05:25:58.659888 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710 [sys.catalog]: This master's current role is: LEADER
I20260928 05:25:58.660986 22389 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:25:58.661243 22366 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:25:58.663054 22366 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:25:58.703192 22366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4de5a5ad7f346efb8f7ab26f411d710: Generated new TSK 0
I20260928 05:25:58.704424 22366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:25:59.392577 22358 catalog_manager.cc:1259] Loaded cluster ID: 8b8be53186834c0aba7c336f5b8c5a61
I20260928 05:25:59.392679 22358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: loading cluster ID for follower catalog manager: success
I20260928 05:25:59.395061 22358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: acquiring CA information for follower catalog manager: success
I20260928 05:25:59.396732 22358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05849924f9864f2d92f858a1b73627ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:25:59.665377 22389 catalog_manager.cc:1259] Loaded cluster ID: 8b8be53186834c0aba7c336f5b8c5a61
I20260928 05:25:59.665479 22389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: loading cluster ID for follower catalog manager: success
I20260928 05:25:59.668112 22389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: acquiring CA information for follower catalog manager: success
I20260928 05:25:59.670046 22389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ae94b6518c941a0b4b5104104b2d96e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:25:59.784951 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22110
I20260928 05:25:59.787032 22172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:25:59.899777 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22110
I20260928 05:25:59.917773 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22184
I20260928 05:25:59.918910 22246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:00.029574 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22184
I20260928 05:26:00.044992 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22258
I20260928 05:26:00.045996 22320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:00.153195 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22258
2026-09-28T05:26:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4462 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:26:00 dist-test-slave-515x krb5kdc[22399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:26:00 dist-test-slave-515x krb5kdc[22399](info): set up 2 sockets
Sep 28 05:26:00 dist-test-slave-515x krb5kdc[22399](info): commencing operation
krb5kdc: starting...
W20260928 05:26:01.161931 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:26:01 dist-test-slave-515x krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:26:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:26:01.371083 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46477
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:01.681386 22412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:01.681689 22412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:01.681823 22412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:01.688740 22412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:01.688846 22412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:01.688922 22412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:01.688993 22412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:01.689054 22412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:01.698527 22412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:01.699770 22412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:01.701023 22412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:01.711812 22418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:01.711774 22420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:01.711858 22417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:01.712733 22412 server_base.cc:1034] running on GCE node
I20260928 05:26:01.713449 22412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:01.714803 22412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:01.715979 22412 hybrid_clock.cc:648] HybridClock initialized: now 1790573161715926 us; error 47 us; skew 500 ppm
I20260928 05:26:01.716301 22412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:01 dist-test-slave-515x krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573161, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:01.719146 22412 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:01.720501 22412 webserver.cc:460] Webserver started at http://127.8.124.254:43693/ using document root <none> and password file <none>
I20260928 05:26:01.721099 22412 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:01.721190 22412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:01.721444 22412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:01.724277 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "493920b8303e4e44a7a49a8c75b9511e"
format_stamp: "Formatted at 2026-09-28 05:26:01 on dist-test-slave-515x"
server_key: "1100c9e8d2d5445181fe9962c7a9fb60"
server_key_iv: "2854e995d2f54fda9dfbd636c0b1859b"
server_key_version: "encryptionkey@0"
I20260928 05:26:01.724946 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "493920b8303e4e44a7a49a8c75b9511e"
format_stamp: "Formatted at 2026-09-28 05:26:01 on dist-test-slave-515x"
server_key: "1100c9e8d2d5445181fe9962c7a9fb60"
server_key_iv: "2854e995d2f54fda9dfbd636c0b1859b"
server_key_version: "encryptionkey@0"
I20260928 05:26:01.729790 22412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 05:26:01.732954 22427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:01.734234 22412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:26:01.734378 22412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "493920b8303e4e44a7a49a8c75b9511e"
format_stamp: "Formatted at 2026-09-28 05:26:01 on dist-test-slave-515x"
server_key: "1100c9e8d2d5445181fe9962c7a9fb60"
server_key_iv: "2854e995d2f54fda9dfbd636c0b1859b"
server_key_version: "encryptionkey@0"
I20260928 05:26:01.734527 22412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:01.761315 22412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:01.764964 22412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:01.765188 22412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:01.781760 22412 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:46477
I20260928 05:26:01.781769 22478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:46477 every 8 connection(s)
I20260928 05:26:01.783015 22412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:01.786554 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:01.788939 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22412
I20260928 05:26:01.789160 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:26:01.790048 8691 external_mini_cluster.cc:1428] Setting key 3b2ae3c2f8ff6e7babd4b348ed83d14a
I20260928 05:26:01.798703 22479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e: Bootstrap starting.
I20260928 05:26:01.802347 22479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:01.803593 22479 log.cc:824] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:26:01 dist-test-slave-515x krb5kdc[22399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:01.806691 22479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e: No bootstrap required, opened a new log
I20260928 05:26:01.813796 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:01.791846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.8.124.254:46477)
0928 05:26:01.792579 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:01.792590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:01.792628 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:26:01.794272 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:01.794294 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:01.794692 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:01.795016 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:01.796759 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:01.798059 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:01.799844 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:01.800530 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:01.800657 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:01.805431 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:01.805438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:01.805452 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:01.805484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:01.809276 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:01.810085 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:01.810110 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:01.810119 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:01.810342 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:01.810685 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:01.810690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:01.810694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:01.811392 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:01.811677 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:01.811970 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 05:26:01.812443 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":334,"threads_started":1}
I20260928 05:26:01.816342 22479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } }
I20260928 05:26:01.816773 22479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:01.816866 22479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493920b8303e4e44a7a49a8c75b9511e, State: Initialized, Role: FOLLOWER
I20260928 05:26:01.817567 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:26:01.817478 22479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } }
I20260928 05:26:01.817794 22479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:01.817914 22479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:01.818064 22479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:01.820996 22479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } }
I20260928 05:26:01.821524 22479 leader_election.cc:302] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493920b8303e4e44a7a49a8c75b9511e; no voters:
I20260928 05:26:01.822050 22479 leader_election.cc:288] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:26:01.822163 22484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:01.822849 22484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [term 1 LEADER]: Becoming Leader. State: Replica: 493920b8303e4e44a7a49a8c75b9511e, State: Running, Role: LEADER
I20260928 05:26:01.823309 22484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } }
I20260928 05:26:01.823930 22479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:01.825878 22486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493920b8303e4e44a7a49a8c75b9511e. Latest consensus state: current_term: 1 leader_uuid: "493920b8303e4e44a7a49a8c75b9511e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } } }
I20260928 05:26:01.826314 22486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:01.827020 22485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493920b8303e4e44a7a49a8c75b9511e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493920b8303e4e44a7a49a8c75b9511e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46477 } } }
I20260928 05:26:01.827298 22485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:01.827725 22493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:01.832345 22493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:01.843546 22493 catalog_manager.cc:1347] Generated new cluster ID: 5620d4a1dee8465596c8382bf8377244
I20260928 05:26:01.843645 22493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:26:01.865394 22493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:26:01.867148 22493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:26:01.883932 22493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493920b8303e4e44a7a49a8c75b9511e: Generated new TSK 0
I20260928 05:26:01.884966 22493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:26:01.930563 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22412
I20260928 05:26:01.931871 22474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:02.026005 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22412
2026-09-28T05:26:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1869 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:26:02 dist-test-slave-515x krb5kdc[22507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:26:02 dist-test-slave-515x krb5kdc[22507](info): set up 2 sockets
Sep 28 05:26:02 dist-test-slave-515x krb5kdc[22507](info): commencing operation
krb5kdc: starting...
W20260928 05:26:03.037892 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:26:03 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:26:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:26:03.248076 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:03.553541 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:03.553835 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:03.553961 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:03.560971 22520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:03.561072 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:03.561143 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:03.561214 22520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:03.561275 22520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:03.570775 22520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:03.572070 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:03.573340 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:03.584173 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:03.584237 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:03.584213 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:03.585127 22520 server_base.cc:1034] running on GCE node
I20260928 05:26:03.585945 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:03.587296 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:03.588505 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790573163588461 us; error 31 us; skew 500 ppm
I20260928 05:26:03.588876 22520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:03 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:03.591833 22520 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:03.593250 22520 webserver.cc:460] Webserver started at http://127.8.124.254:42227/ using document root <none> and password file <none>
I20260928 05:26:03.593963 22520 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:03.594076 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:03.594349 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:03.597195 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "83b2a4438a1e4b258b2acff11f8e0653"
format_stamp: "Formatted at 2026-09-28 05:26:03 on dist-test-slave-515x"
server_key: "34422b46817bc13f997e122c8ed71476"
server_key_iv: "f19d4f12e112c05444e6bbec9ace01a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:03.597952 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "83b2a4438a1e4b258b2acff11f8e0653"
format_stamp: "Formatted at 2026-09-28 05:26:03 on dist-test-slave-515x"
server_key: "34422b46817bc13f997e122c8ed71476"
server_key_iv: "f19d4f12e112c05444e6bbec9ace01a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:03.602860 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:26:03.605971 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:03.607184 22520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 05:26:03.607335 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "83b2a4438a1e4b258b2acff11f8e0653"
format_stamp: "Formatted at 2026-09-28 05:26:03 on dist-test-slave-515x"
server_key: "34422b46817bc13f997e122c8ed71476"
server_key_iv: "f19d4f12e112c05444e6bbec9ace01a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:03.607502 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:03.632668 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:03.636940 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:03.637207 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:03.653868 22520 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45313
I20260928 05:26:03.653891 22586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45313 every 8 connection(s)
I20260928 05:26:03.655145 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:03.655726 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22520
I20260928 05:26:03.655921 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:26:03.656667 8691 external_mini_cluster.cc:1428] Setting key 1e68016cab51eb15b3543806a4fd3e5c
I20260928 05:26:03.659353 22587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:26:03 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:03.671413 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap starting.
I20260928 05:26:03.674520 22587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:03.675685 22587 log.cc:824] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:03.678719 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: No bootstrap required, opened a new log
I20260928 05:26:03.678998 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:03.658562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.8.124.254:45313)
0928 05:26:03.659354 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:03.659369 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:03.659419 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:26:03.661580 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:03.661633 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:03.662109 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:03.662552 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:03.663675 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:03.665711 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:03.666970 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:03.667932 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:03.668101 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:03.672042 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:03.672049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:03.672063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:03.672101 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:03.674895 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:03.675656 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:03.675662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:03.675667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:03.675855 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:03.676279 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:03.676287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:03.676290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:03.676999 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:03.677210 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:03.677671 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 05:26:03.678021 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":304,"threads_started":1}
I20260928 05:26:03.686712 22587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:03.687023 22587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:03.687114 22587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Initialized, Role: FOLLOWER
I20260928 05:26:03.687637 22587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:03.687853 22587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:03.687970 22587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:03.688114 22587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:03.690239 22587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:03.690770 22587 leader_election.cc:302] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b2a4438a1e4b258b2acff11f8e0653; no voters:
I20260928 05:26:03.691373 22587 leader_election.cc:288] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:26:03.691457 22592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:03.692088 22592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 LEADER]: Becoming Leader. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Running, Role: LEADER
I20260928 05:26:03.692591 22592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:03.693228 22587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:03.694968 22594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b2a4438a1e4b258b2acff11f8e0653. Latest consensus state: current_term: 1 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:03.694945 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:03.695359 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:03.695869 22594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:03.696265 22597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:03.701352 22597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:03.710911 22597 catalog_manager.cc:1347] Generated new cluster ID: 5eaadf5a064347e7b9e4ef9480b6e5d7
I20260928 05:26:03.711082 22597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:26:03.731067 22597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:26:03.732362 22597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:26:03.740281 22597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Generated new TSK 0
I20260928 05:26:03.741001 22597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:26:03.799273 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:04.136238 22613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:04.136511 22613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:04.136639 22613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:04.143761 22613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:04.143858 22613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:04.144001 22613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:26:04.153584 22613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:04.154836 22613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:04.156108 22613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:04.168123 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:04.168186 22619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:04.168200 22618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:04.168830 22613 server_base.cc:1034] running on GCE node
I20260928 05:26:04.169377 22613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:04.170291 22613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:04.171473 22613 hybrid_clock.cc:648] HybridClock initialized: now 1790573164171414 us; error 53 us; skew 500 ppm
I20260928 05:26:04.171759 22613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:04 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573164, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:04.174630 22613 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:26:04.175982 22613 webserver.cc:460] Webserver started at http://127.8.124.193:44347/ using document root <none> and password file <none>
I20260928 05:26:04.176616 22613 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:04.176723 22613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:04.176990 22613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:04.180102 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "4f40593a597e43dc9b3a6e5086c9474f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "8613dbd0efc2adac9982d11549f4dd19"
server_key_iv: "1bd371e9f736b2e418edb1674789aa6c"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.180825 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "4f40593a597e43dc9b3a6e5086c9474f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "8613dbd0efc2adac9982d11549f4dd19"
server_key_iv: "1bd371e9f736b2e418edb1674789aa6c"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.185644 22613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 05:26:04.188784 22628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.190055 22613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:26:04.190210 22613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "4f40593a597e43dc9b3a6e5086c9474f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "8613dbd0efc2adac9982d11549f4dd19"
server_key_iv: "1bd371e9f736b2e418edb1674789aa6c"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.190385 22613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:04.220583 22613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:04.224500 22613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:04.224762 22613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:04.225704 22613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:04.226999 22613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:04.227116 22613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.227265 22613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:04.227337 22613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.256434 22613 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:41847
I20260928 05:26:04.256470 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:41847 every 8 connection(s)
I20260928 05:26:04.257706 22613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:26:04.262763 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22613
I20260928 05:26:04.262945 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:26:04.263590 8691 external_mini_cluster.cc:1428] Setting key ac39f1fac5e88786b3a8fb3f63def733
Sep 28 05:26:04 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573164, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:04.276331 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:04.260088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:35589 (local address 127.8.124.254:45313)
0928 05:26:04.260492 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:04.260497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:04.261374 (+ 877us) server_negotiation.cc:407] Connection header received
0928 05:26:04.263248 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:04.263254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:04.263340 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:04.263435 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:04.264912 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.265637 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.267130 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.267520 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.267624 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:04.271012 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:04.271017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:04.271020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:04.271051 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:04.273476 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:04.274290 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:04.274294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:04.274296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:04.274411 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:04.274825 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:04.274828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:04.274830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:04.275066 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:04.275169 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:04.275917 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 05:26:04.276083 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":194,"threads_started":1}
I20260928 05:26:04.277681 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:04.260554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:35589)
0928 05:26:04.261211 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:04.261247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:04.262965 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:04.263694 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:04.263705 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:04.264197 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:04.264742 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:04.264760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.265778 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.265782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:04.266992 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:04.267004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.267723 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.267730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:04.267954 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:04.268631 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:04.268654 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:04.270798 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:04.273634 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:04.273642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:04.273660 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:04.274197 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:04.274496 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:04.274501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:04.274504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:04.274737 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:04.275169 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:04.275176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:04.275566 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 05:26:04.275836 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 05:26:04.276344 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260928 05:26:04.279126 22741 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:04.279421 22741 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:04.280078 22741 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:04.282080 22552 ts_manager.cc:194] Registered new tserver with Master: 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:04.283456 22552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:35589
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:26:04.308667 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:04.612771 22747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:04.613062 22747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:04.613198 22747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:04.620307 22747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:04.620419 22747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:04.620572 22747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:26:04.629953 22747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:04.631168 22747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:04.632411 22747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:04.644454 22753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:04.644518 22752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:04.644483 22755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:04.645780 22747 server_base.cc:1034] running on GCE node
I20260928 05:26:04.646294 22747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:04.647110 22747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:04.648294 22747 hybrid_clock.cc:648] HybridClock initialized: now 1790573164648258 us; error 37 us; skew 500 ppm
I20260928 05:26:04.648577 22747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:04 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573164, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:04.651507 22747 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:26:04.652804 22747 webserver.cc:460] Webserver started at http://127.8.124.194:34737/ using document root <none> and password file <none>
I20260928 05:26:04.653484 22747 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:04.653582 22747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:04.653895 22747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:04.656642 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "f4f9b5b5bd07425ca2105cc673af192f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "c363b27a245982df5369c0a19b890234"
server_key_iv: "17057f79b43e1c7f686a870dc96f66a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.657332 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "f4f9b5b5bd07425ca2105cc673af192f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "c363b27a245982df5369c0a19b890234"
server_key_iv: "17057f79b43e1c7f686a870dc96f66a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.662317 22747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:26:04.665619 22762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.666929 22747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:04.667070 22747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "f4f9b5b5bd07425ca2105cc673af192f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "c363b27a245982df5369c0a19b890234"
server_key_iv: "17057f79b43e1c7f686a870dc96f66a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:04.667235 22747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:04.685920 22747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:04.689983 22747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:04.690248 22747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:04.691126 22747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:04.692360 22747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:04.692437 22747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.692551 22747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:04.692601 22747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:04.721122 22747 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:42649
I20260928 05:26:04.721158 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:42649 every 8 connection(s)
I20260928 05:26:04.722373 22747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:26:04.727214 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22747
I20260928 05:26:04.727386 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:26:04.727819 8691 external_mini_cluster.cc:1428] Setting key e94998500e73a8f57943ea8bb1a3281e
Sep 28 05:26:04 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573164, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:04.741184 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:04.724854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:40617 (local address 127.8.124.254:45313)
0928 05:26:04.725034 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:04.725039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:04.726209 (+ 1170us) server_negotiation.cc:407] Connection header received
0928 05:26:04.728032 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:04.728037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:04.728118 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:04.728204 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:04.729917 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.730630 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.732079 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.732476 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.732581 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:04.735888 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:04.735894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:04.735900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:04.735940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:04.738349 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:04.739134 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:04.739138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:04.739140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:04.739260 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:04.739645 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:04.739647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:04.739649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:04.739898 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:04.740001 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:04.740765 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 05:26:04.740929 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:26:04.742439 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:04.725381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:40617)
0928 05:26:04.726062 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:04.726098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:04.727732 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:04.728462 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:04.728474 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:04.729057 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:04.729762 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:04.729781 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.730771 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.730775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:04.731947 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:04.731963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:04.732678 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:04.732686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:04.732891 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:04.733507 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:04.733531 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:04.735666 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:04.738475 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:04.738484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:04.738498 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:04.739019 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:04.739348 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:04.739352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:04.739355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:04.739559 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:04.739996 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:04.740003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:04.740376 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 05:26:04.740671 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 05:26:04.741165 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":273,"threads_started":1}
I20260928 05:26:04.743788 22875 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:04.744058 22875 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:04.744742 22875 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:04.746170 22552 ts_manager.cc:194] Registered new tserver with Master: f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649)
I20260928 05:26:04.747241 22552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:40617
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:26:04.774015 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:05.081007 22880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:05.081267 22880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:05.081372 22880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:05.088325 22880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:05.088424 22880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:05.088541 22880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:26:05.098008 22880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:05.099176 22880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:05.100445 22880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:05.112911 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:05.112919 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:05.112846 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:05.113787 22880 server_base.cc:1034] running on GCE node
I20260928 05:26:05.114385 22880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:05.115275 22880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:05.116477 22880 hybrid_clock.cc:648] HybridClock initialized: now 1790573165116425 us; error 45 us; skew 500 ppm
I20260928 05:26:05.116811 22880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:05 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573165, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:05.119895 22880 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:26:05.121258 22880 webserver.cc:460] Webserver started at http://127.8.124.195:38895/ using document root <none> and password file <none>
I20260928 05:26:05.121985 22880 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:05.122098 22880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:05.122367 22880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:05.125299 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "6ba075edfa804045986ea9100879a9fe"
format_stamp: "Formatted at 2026-09-28 05:26:05 on dist-test-slave-515x"
server_key: "ab26a0b13f291395d8a6dcc79acd7479"
server_key_iv: "153e6007c5888b0a1e2aca47e93539ec"
server_key_version: "encryptionkey@0"
I20260928 05:26:05.126157 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "6ba075edfa804045986ea9100879a9fe"
format_stamp: "Formatted at 2026-09-28 05:26:05 on dist-test-slave-515x"
server_key: "ab26a0b13f291395d8a6dcc79acd7479"
server_key_iv: "153e6007c5888b0a1e2aca47e93539ec"
server_key_version: "encryptionkey@0"
I20260928 05:26:05.131289 22880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:26:05.134683 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:05.136058 22880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:26:05.136212 22880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "6ba075edfa804045986ea9100879a9fe"
format_stamp: "Formatted at 2026-09-28 05:26:05 on dist-test-slave-515x"
server_key: "ab26a0b13f291395d8a6dcc79acd7479"
server_key_iv: "153e6007c5888b0a1e2aca47e93539ec"
server_key_version: "encryptionkey@0"
I20260928 05:26:05.136397 22880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:05.159441 22880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:05.163233 22880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:05.163489 22880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:05.164407 22880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:05.165766 22880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:05.165853 22880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:05.165967 22880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:05.166018 22880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:05.196939 22880 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:44927
I20260928 05:26:05.196961 23007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:44927 every 8 connection(s)
I20260928 05:26:05.198235 22880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:26:05.204734 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 22880
I20260928 05:26:05.204907 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:26:05.205339 8691 external_mini_cluster.cc:1428] Setting key 810c8a9b150339bff28cf6edb0e75e53
Sep 28 05:26:05 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573165, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:05.216792 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.200560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:58473 (local address 127.8.124.254:45313)
0928 05:26:05.200738 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.200742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.201990 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 05:26:05.203776 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.203782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.203887 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.203977 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:05.205546 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.206366 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.207882 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.208289 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.208401 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.211688 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:05.211693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:05.211696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:05.211728 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:05.214009 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.214680 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.214685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.214689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.214801 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.215177 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.215182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.215186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.215449 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:05.215590 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.216238 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.216458 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:26:05.217756 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.201097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:58473)
0928 05:26:05.201826 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.201869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.203501 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.204216 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.204224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.204718 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:05.205340 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.205358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.206492 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.206497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.207739 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.207752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.208503 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.208510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.208731 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.209298 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:05.209321 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:05.211470 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:05.214126 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:05.214135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:05.214152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:05.214574 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:05.214898 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:05.214903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:05.214907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:05.215076 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:05.215565 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:05.215572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:05.215897 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.216129 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.216633 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":264,"threads_started":1}
I20260928 05:26:05.219079 23008 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:05.219404 23008 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:05.220070 23008 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:05.221386 22552 ts_manager.cc:194] Registered new tserver with Master: 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927)
I20260928 05:26:05.222187 22552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:58473
I20260928 05:26:05.232045 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:26:05.248096 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.236408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.8.124.254:45313)
0928 05:26:05.236599 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.236605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.236937 (+ 332us) server_negotiation.cc:407] Connection header received
0928 05:26:05.237092 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.237098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.237166 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.237243 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:05.238593 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.239291 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.240761 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.241175 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.241280 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.243409 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:05.243413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:05.243416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:05.243446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:05.245728 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.246315 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.246321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.246324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.246448 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.246762 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.246767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.246770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.247082 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:05.247297 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.247597 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.247797 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:26:05.251619 22552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:26:05.258208 22552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:26:05.286360 22741 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:05.291833 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.269775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.8.124.194:42649)
0928 05:26:05.282613 (+ 12838us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.282622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.282653 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:26:05.282753 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.282760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.283048 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.283265 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:05.284266 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.286003 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.288773 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.290451 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.290568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.290770 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.291338 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.291504 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12671,"thread_start_us":12833,"threads_started":1}
I20260928 05:26:05.291858 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.269578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:42649 (local address 127.0.0.1:44084)
0928 05:26:05.282230 (+ 12652us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.282286 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.282488 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.283577 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.283587 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.283624 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:05.284052 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.284067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.286218 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.286224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.288566 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.288583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.290949 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.290955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.291127 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.291161 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.291271 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.291472 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12346,"thread_start_us":12149,"threads_started":1}
I20260928 05:26:05.295883 22810 tablet_service.cc:1467] Processing CreateTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 (DEFAULT_TABLE table=test-table [id=d9d64e40883140f1826131a908b72528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:05.297376 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71358ec5ce2849b198c156d7b866b8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:05.298565 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.270247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.8.124.195:44927)
0928 05:26:05.290376 (+ 20129us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.290384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.290418 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:26:05.290978 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.290987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.291262 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.291457 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:05.292442 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.293924 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.296284 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.297582 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.297719 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.297941 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.298128 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.298276 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19949,"thread_start_us":20588,"threads_started":1}
I20260928 05:26:05.298600 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.269647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:44927 (local address 127.0.0.1:47038)
0928 05:26:05.289405 (+ 19758us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.289425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.289555 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.291703 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.291708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.291757 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:05.292197 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.292208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.294044 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.294050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.296115 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.296130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.297784 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.297790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.297951 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.297985 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.298077 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.298207 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19619,"mutex_wait_us":5,"spinlock_wait_cycles":8320,"thread_start_us":19453,"threads_started":1}
I20260928 05:26:05.299747 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.270127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52634)
0928 05:26:05.289708 (+ 19581us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.289723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.289831 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.291702 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.291708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.291764 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:05.292443 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.292454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.294924 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.294929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.296904 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.296915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.299042 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.299046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.299155 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.299186 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.299261 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.299380 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19413,"mutex_wait_us":240,"spinlock_wait_cycles":10624,"thread_start_us":19483,"threads_started":1}
I20260928 05:26:05.299813 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.270458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.8.124.193:41847)
0928 05:26:05.290924 (+ 20466us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.290931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.290956 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:26:05.291031 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.291036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.291262 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.291456 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:05.292607 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.294782 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.297072 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.298584 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.298714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.298932 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.299325 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.299485 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20297,"thread_start_us":20141,"threads_started":1}
I20260928 05:26:05.302258 22943 tablet_service.cc:1467] Processing CreateTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 (DEFAULT_TABLE table=test-table [id=d9d64e40883140f1826131a908b72528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:05.302848 22676 tablet_service.cc:1467] Processing CreateTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 (DEFAULT_TABLE table=test-table [id=d9d64e40883140f1826131a908b72528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:05.303627 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71358ec5ce2849b198c156d7b866b8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:05.303938 22676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71358ec5ce2849b198c156d7b866b8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:05.313163 23026 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Bootstrap starting.
I20260928 05:26:05.315750 23027 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Bootstrap starting.
I20260928 05:26:05.316721 23026 tablet_bootstrap.cc:654] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:05.317747 23026 log.cc:824] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:05.318331 23028 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Bootstrap starting.
I20260928 05:26:05.319176 23027 tablet_bootstrap.cc:654] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:05.320169 23027 log.cc:824] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:05.321511 23026 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: No bootstrap required, opened a new log
I20260928 05:26:05.321841 23026 ts_tablet_manager.cc:1397] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 05:26:05.322232 23028 tablet_bootstrap.cc:654] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:05.323354 23027 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: No bootstrap required, opened a new log
I20260928 05:26:05.323390 23028 log.cc:824] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:05.323586 23027 ts_tablet_manager.cc:1397] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:26:05.326224 23028 tablet_bootstrap.cc:492] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: No bootstrap required, opened a new log
I20260928 05:26:05.326459 23028 ts_tablet_manager.cc:1397] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 05:26:05.330899 23027 raft_consensus.cc:348] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.331228 23027 raft_consensus.cc:374] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:05.331343 23027 raft_consensus.cc:729] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f40593a597e43dc9b3a6e5086c9474f, State: Initialized, Role: FOLLOWER
I20260928 05:26:05.331756 23026 raft_consensus.cc:348] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.331907 23027 consensus_queue.cc:260] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.332155 23026 raft_consensus.cc:374] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:05.332274 23026 raft_consensus.cc:729] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9b5b5bd07425ca2105cc673af192f, State: Initialized, Role: FOLLOWER
I20260928 05:26:05.332911 23026 consensus_queue.cc:260] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.334084 22875 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:05.335999 23026 ts_tablet_manager.cc:1428] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 05:26:05.338652 23028 raft_consensus.cc:348] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.339116 23028 raft_consensus.cc:374] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:05.339244 23028 raft_consensus.cc:729] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba075edfa804045986ea9100879a9fe, State: Initialized, Role: FOLLOWER
I20260928 05:26:05.339810 23028 consensus_queue.cc:260] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.340824 23008 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:05.341449 23028 ts_tablet_manager.cc:1428] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260928 05:26:05.341475 23027 ts_tablet_manager.cc:1428] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260928 05:26:05.427428 23033 raft_consensus.cc:482] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:26:05.427721 23033 raft_consensus.cc:504] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.429486 23033 leader_election.cc:288] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927), 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:05.435781 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.430042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:51273 (local address 127.8.124.195:44927)
0928 05:26:05.430366 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.430372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.430733 (+ 361us) server_negotiation.cc:407] Connection header received
0928 05:26:05.430931 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.430934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.430993 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.431063 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:05.431577 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.432281 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.434091 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.434932 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.435078 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.435153 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.435420 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.435538 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260928 05:26:05.435930 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.429944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:49897 (local address 127.8.124.193:41847)
0928 05:26:05.430185 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.430190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.430941 (+ 751us) server_negotiation.cc:407] Connection header received
0928 05:26:05.431092 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.431096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.431158 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.431228 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:05.431778 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.432492 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.434277 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.435077 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.435220 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.435288 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.435577 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.435709 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:26:05.435956 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.429828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:44927 (local address 127.8.124.194:51273)
0928 05:26:05.430629 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.430645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.430793 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.431094 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.431098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.431121 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:05.431448 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.431458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.432411 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.432415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.433956 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.433968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.435122 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.435128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.435269 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.435304 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.435395 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.435518 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":2,"thread_start_us":465,"threads_started":1}
I20260928 05:26:05.436077 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.430086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.8.124.194:49897)
0928 05:26:05.430832 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:05.430846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:05.430966 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:05.431266 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:05.431271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:05.431294 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:05.431633 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.431641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.432622 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.432627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.434127 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:05.434142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.435270 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.435276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:05.435439 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.435463 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:26:05.435547 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:26:05.435666 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":2,"thread_start_us":430,"threads_started":1}
I20260928 05:26:05.437116 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71358ec5ce2849b198c156d7b866b8c9" candidate_uuid: "f4f9b5b5bd07425ca2105cc673af192f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba075edfa804045986ea9100879a9fe" is_pre_election: true
I20260928 05:26:05.437197 22696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71358ec5ce2849b198c156d7b866b8c9" candidate_uuid: "f4f9b5b5bd07425ca2105cc673af192f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f40593a597e43dc9b3a6e5086c9474f" is_pre_election: true
I20260928 05:26:05.437753 22963 raft_consensus.cc:2454] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9b5b5bd07425ca2105cc673af192f in term 0.
I20260928 05:26:05.437753 22696 raft_consensus.cc:2454] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f9b5b5bd07425ca2105cc673af192f in term 0.
I20260928 05:26:05.438403 22764 leader_election.cc:302] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f40593a597e43dc9b3a6e5086c9474f, f4f9b5b5bd07425ca2105cc673af192f; no voters:
I20260928 05:26:05.438733 23033 raft_consensus.cc:2781] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:26:05.438867 23033 raft_consensus.cc:482] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:26:05.438962 23033 raft_consensus.cc:3037] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:05.441437 23033 raft_consensus.cc:504] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:05.442006 23033 leader_election.cc:288] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [CANDIDATE]: Term 1 election: Requested vote from peers 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927), 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:05.442445 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71358ec5ce2849b198c156d7b866b8c9" candidate_uuid: "f4f9b5b5bd07425ca2105cc673af192f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba075edfa804045986ea9100879a9fe"
I20260928 05:26:05.442490 22696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71358ec5ce2849b198c156d7b866b8c9" candidate_uuid: "f4f9b5b5bd07425ca2105cc673af192f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f40593a597e43dc9b3a6e5086c9474f"
I20260928 05:26:05.442657 22696 raft_consensus.cc:3037] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:05.442633 22963 raft_consensus.cc:3037] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:05.444694 22696 raft_consensus.cc:2454] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9b5b5bd07425ca2105cc673af192f in term 1.
I20260928 05:26:05.444839 22963 raft_consensus.cc:2454] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f9b5b5bd07425ca2105cc673af192f in term 1.
I20260928 05:26:05.445053 22764 leader_election.cc:302] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f40593a597e43dc9b3a6e5086c9474f, f4f9b5b5bd07425ca2105cc673af192f; no voters:
I20260928 05:26:05.445305 23033 raft_consensus.cc:2781] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:05.445740 23033 raft_consensus.cc:686] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 1 LEADER]: Becoming Leader. State: Replica: f4f9b5b5bd07425ca2105cc673af192f, State: Running, Role: LEADER
I20260928 05:26:05.446242 23033 consensus_queue.cc:237] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
W20260928 05:26:05.449630 23009 tablet.cc:2369] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:26:05.450608 22552 catalog_manager.cc:5495] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194). New cstate: current_term: 1 leader_uuid: "f4f9b5b5bd07425ca2105cc673af192f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } health_report { overall_health: UNKNOWN } } }
W20260928 05:26:05.474221 22876 tablet.cc:2369] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:26:05.489236 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.484084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.8.124.194:42649)
0928 05:26:05.484294 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.484298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.484315 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:26:05.484527 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.484532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.484620 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.484753 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:05.485073 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.485886 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.487263 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.487666 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.487758 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.488230 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:05.488729 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:26:05.488834 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.488886 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.488950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:26:05.496826 22696 raft_consensus.cc:1261] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Refusing update from remote peer f4f9b5b5bd07425ca2105cc673af192f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:05.496958 22963 raft_consensus.cc:1261] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Refusing update from remote peer f4f9b5b5bd07425ca2105cc673af192f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:05.497727 23037 consensus_queue.cc:1035] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:05.498096 23033 consensus_queue.cc:1035] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:05.508391 23040 mvcc.cc:205] Tried to move back new op lower bound from 7334187685866393600 to 7334187685671768064. Current Snapshot: MvccSnapshot[applied={T|T < 7334187685866393600}]
I20260928 05:26:05.511476 23041 mvcc.cc:205] Tried to move back new op lower bound from 7334187685866393600 to 7334187685671768064. Current Snapshot: MvccSnapshot[applied={T|T < 7334187685866393600}]
I20260928 05:26:05.521212 22551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:05.521487 22551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:05.525002 22551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Sending DeleteTablet for 3 replicas of tablet 71358ec5ce2849b198c156d7b866b8c9
I20260928 05:26:05.526028 22810 tablet_service.cc:1514] Processing DeleteTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44084
I20260928 05:26:05.526075 22943 tablet_service.cc:1514] Processing DeleteTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:47038
I20260928 05:26:05.526161 22676 tablet_service.cc:1514] Processing DeleteTablet for tablet 71358ec5ce2849b198c156d7b866b8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:05 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:52634
I20260928 05:26:05.530544 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22613
I20260928 05:26:05.537096 23050 tablet_replica.cc:330] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: stopping tablet replica
I20260928 05:26:05.538228 23050 raft_consensus.cc:2229] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:05.539458 23050 raft_consensus.cc:2258] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:05.540501 23051 tablet_replica.cc:330] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: stopping tablet replica
I20260928 05:26:05.541438 23051 raft_consensus.cc:2229] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:26:05.542263 23051 raft_consensus.cc:2258] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:05.543808 23050 ts_tablet_manager.cc:1905] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:05.545989 23051 ts_tablet_manager.cc:1905] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:05.549160 23049 tablet_replica.cc:330] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: stopping tablet replica
I20260928 05:26:05.549696 23049 raft_consensus.cc:2229] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:05.550933 23050 ts_tablet_manager.cc:1918] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:05.551151 23050 log.cc:1196] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/71358ec5ce2849b198c156d7b866b8c9
I20260928 05:26:05.551724 23050 ts_tablet_manager.cc:1939] T 71358ec5ce2849b198c156d7b866b8c9 P 4f40593a597e43dc9b3a6e5086c9474f: Deleting consensus metadata
I20260928 05:26:05.552085 23049 raft_consensus.cc:2258] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:05.553236 23051 ts_tablet_manager.cc:1918] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:05.553458 23051 log.cc:1196] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/71358ec5ce2849b198c156d7b866b8c9
I20260928 05:26:05.553699 22537 catalog_manager.cc:4841] TS 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847): tablet 71358ec5ce2849b198c156d7b866b8c9 (table test-table [id=d9d64e40883140f1826131a908b72528]) successfully deleted
I20260928 05:26:05.554047 23051 ts_tablet_manager.cc:1939] T 71358ec5ce2849b198c156d7b866b8c9 P f4f9b5b5bd07425ca2105cc673af192f: Deleting consensus metadata
Sep 28 05:26:05 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:26:05.556015 23049 ts_tablet_manager.cc:1905] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:05.555996 22536 catalog_manager.cc:4841] TS f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649): tablet 71358ec5ce2849b198c156d7b866b8c9 (table test-table [id=d9d64e40883140f1826131a908b72528]) successfully deleted
I20260928 05:26:05.561115 23049 ts_tablet_manager.cc:1918] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:05.561285 23049 log.cc:1196] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/71358ec5ce2849b198c156d7b866b8c9
I20260928 05:26:05.561767 23049 ts_tablet_manager.cc:1939] T 71358ec5ce2849b198c156d7b866b8c9 P 6ba075edfa804045986ea9100879a9fe: Deleting consensus metadata
I20260928 05:26:05.563382 22539 catalog_manager.cc:4841] TS 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927): tablet 71358ec5ce2849b198c156d7b866b8c9 (table test-table [id=d9d64e40883140f1826131a908b72528]) successfully deleted
I20260928 05:26:05.563818 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.531791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.8.124.193:41847)
0928 05:26:05.531971 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.531978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.543174 (+ 11196us) server_negotiation.cc:407] Connection header received
0928 05:26:05.543643 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.543649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.543727 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.543810 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:05.545725 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.546827 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.549924 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.550591 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.550696 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.556890 (+ 6194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:05.556906 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:05.556925 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:05.556968 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:05.560593 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.561521 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.561531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.561534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.561713 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.562140 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.562145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.562149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.562606 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:05.562927 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.563215 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.563442 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:26:05.564563 22736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:05.652715 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22613
I20260928 05:26:05.667495 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22747
Sep 28 05:26:05 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:26:05.682459 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.668205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.8.124.194:42649)
0928 05:26:05.668380 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.668387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.668516 (+ 129us) server_negotiation.cc:407] Connection header received
0928 05:26:05.668690 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.668696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.668765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.668847 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:05.670207 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.671065 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.672511 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.672935 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.673037 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.677066 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:05.677079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:05.677095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:05.677132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:05.679875 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.680513 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.680520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.680523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.680673 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.681013 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.681018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.681020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.681372 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:05.681716 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.681937 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.682135 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:26:05.683218 22870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:05.794814 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22747
I20260928 05:26:05.809340 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22880
Sep 28 05:26:05 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:26:05.824386 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:05.810020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.8.124.195:44927)
0928 05:26:05.810209 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:05.810214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:05.810306 (+ 92us) server_negotiation.cc:407] Connection header received
0928 05:26:05.810455 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:05.810458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:05.810521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:05.810600 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:05.811976 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.812671 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.814105 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:05.814480 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:05.814570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:05.818543 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:05.818556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:05.818571 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:05.818608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:05.821591 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.822361 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.822370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.822374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.822533 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:05.822936 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:05.822942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:05.822946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:05.823292 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:05.823555 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:05.823855 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 05:26:05.824052 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:26:05.824935 23003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:05.910672 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22880
I20260928 05:26:05.923600 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 22520
I20260928 05:26:05.924675 22582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:06.033371 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 22520
I20260928 05:26:06.050155 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--webserver_interface=127.8.124.254
--webserver_port=42227
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:06.361343 23057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:06.361632 23057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:06.361763 23057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:06.368713 23057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:06.368814 23057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:06.368870 23057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:06.368923 23057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:06.368965 23057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:06.378458 23057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=42227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:06.379609 23057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:06.380837 23057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:06.400802 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:06.400892 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:06.400914 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:06.401539 23057 server_base.cc:1034] running on GCE node
I20260928 05:26:06.402374 23057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:06.403872 23057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:06.405112 23057 hybrid_clock.cc:648] HybridClock initialized: now 1790573166405048 us; error 42 us; skew 500 ppm
I20260928 05:26:06.405537 23057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:06 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573166, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:06.408794 23057 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:06.410418 23057 webserver.cc:460] Webserver started at http://127.8.124.254:42227/ using document root <none> and password file <none>
I20260928 05:26:06.411127 23057 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:06.411254 23057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:06.416990 23057 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:26:06.419756 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:06.420964 23057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:26:06.421121 23057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "83b2a4438a1e4b258b2acff11f8e0653"
format_stamp: "Formatted at 2026-09-28 05:26:03 on dist-test-slave-515x"
server_key: "34422b46817bc13f997e122c8ed71476"
server_key_iv: "f19d4f12e112c05444e6bbec9ace01a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:06.422155 23057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:06.436280 23057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:06.439929 23057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:06.440151 23057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:06.456876 23057 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45313
I20260928 05:26:06.456897 23124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45313 every 8 connection(s)
I20260928 05:26:06.458364 23057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:06.458945 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23057
I20260928 05:26:06.459867 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:41847
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=44347
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:26:06.465332 23125 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:26:06.469372 23125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap starting.
I20260928 05:26:06.496414 23125 log.cc:824] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:06.510972 23125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:26:06.511547 23125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap complete.
I20260928 05:26:06.524972 23125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:06.526201 23125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Initialized, Role: FOLLOWER
I20260928 05:26:06.526911 23125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:06.527177 23125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:06.527293 23125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:06.527468 23125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:26:06.532758 23125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:06.533402 23125 leader_election.cc:302] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b2a4438a1e4b258b2acff11f8e0653; no voters:
I20260928 05:26:06.534034 23125 leader_election.cc:288] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:26:06.534179 23129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:26:06.535359 23129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 LEADER]: Becoming Leader. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Running, Role: LEADER
I20260928 05:26:06.535872 23129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:06.536150 23125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:06.538066 23131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b2a4438a1e4b258b2acff11f8e0653. Latest consensus state: current_term: 2 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:06.538408 23131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:06.538921 23130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:06.539175 23130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:06.548516 23135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:06.560354 23135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:06.561792 23135 catalog_manager.cc:1259] Loaded cluster ID: 5eaadf5a064347e7b9e4ef9480b6e5d7
I20260928 05:26:06.561964 23135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:26:06.564888 23135 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 05:26:06.581362 23135 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 05:26:06.590725 23135 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:26:06.592331 23135 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Loaded TSK: 0
I20260928 05:26:06.593803 23135 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:26:06.813969 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:06.814251 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:06.814385 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:06.821336 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:06.821444 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:06.821590 23127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:26:06.831897 23127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:41847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=44347
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:06.833150 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:06.834518 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:06.847262 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:06.847242 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:06.847920 23127 server_base.cc:1034] running on GCE node
W20260928 05:26:06.848177 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:06.848790 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:06.849718 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:06.850934 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790573166850887 us; error 38 us; skew 500 ppm
I20260928 05:26:06.851238 23127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:06 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573166, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:06.854365 23127 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:26:06.855751 23127 webserver.cc:460] Webserver started at http://127.8.124.193:44347/ using document root <none> and password file <none>
I20260928 05:26:06.856531 23127 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:06.856693 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:06.862447 23127 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 05:26:06.865165 23162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:06.866467 23127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:06.866600 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "4f40593a597e43dc9b3a6e5086c9474f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "8613dbd0efc2adac9982d11549f4dd19"
server_key_iv: "1bd371e9f736b2e418edb1674789aa6c"
server_key_version: "encryptionkey@0"
I20260928 05:26:06.867491 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:06.887939 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:06.892150 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:06.892405 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:06.893446 23127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:06.894776 23127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:06.894869 23127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:06.894980 23127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:06.895048 23127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:06.924690 23127 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:41847
I20260928 05:26:06.924772 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:41847 every 8 connection(s)
I20260928 05:26:06.926178 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:26:06.933820 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23127
I20260928 05:26:06.934517 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:42649
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=34737
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:26:06 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573166, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:06.948722 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:06.928792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:42717 (local address 127.8.124.254:45313)
0928 05:26:06.929305 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:06.929317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:06.929993 (+ 676us) server_negotiation.cc:407] Connection header received
0928 05:26:06.932013 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:06.932025 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:06.932359 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:06.932659 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:06.934119 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:06.936032 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:06.937545 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:06.938514 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:06.938669 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:06.941917 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:06.941925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:06.941937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:06.941971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:06.944580 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:06.945250 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:06.945257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:06.945262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:06.945411 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:06.945847 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:06.945850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:06.945852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:06.946287 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:06.946454 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:06.947173 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 05:26:06.947541 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260928 05:26:06.948714 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:06.929056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:42717)
0928 05:26:06.929841 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:06.929890 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:06.931580 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:06.932804 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:06.932814 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:06.933290 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:06.933908 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:06.933932 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:06.936170 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:06.936174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:06.937429 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:06.937441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:06.938701 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:06.938707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:06.938888 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:06.939477 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:06.939503 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:06.941675 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:06.944707 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:06.944715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:06.944729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:06.945115 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:06.945513 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:06.945518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:06.945522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:06.945736 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:06.946409 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:06.946417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:06.946725 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 05:26:06.946954 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 05:26:06.947454 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260928 05:26:06.951726 23275 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:06.951985 23275 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:06.952647 23275 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:06.954949 23090 ts_manager.cc:194] Registered new tserver with Master: 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:06.957044 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:42717
W20260928 05:26:07.250322 23279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:07.250586 23279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:07.250715 23279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:07.257858 23279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:07.257967 23279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:07.258165 23279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:26:07.268311 23279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:42649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=34737
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:07.269526 23279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:07.270965 23279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:07.283484 23284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:07.283421 23285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:07.283452 23287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:07.284349 23279 server_base.cc:1034] running on GCE node
I20260928 05:26:07.284889 23279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:07.285800 23279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:07.286994 23279 hybrid_clock.cc:648] HybridClock initialized: now 1790573167286941 us; error 43 us; skew 500 ppm
I20260928 05:26:07.287297 23279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:07 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573167, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:07.290297 23279 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:26:07.291683 23279 webserver.cc:460] Webserver started at http://127.8.124.194:34737/ using document root <none> and password file <none>
I20260928 05:26:07.292335 23279 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:07.292440 23279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:07.297551 23279 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:07.300112 23294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.301383 23279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:26:07.301505 23279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "f4f9b5b5bd07425ca2105cc673af192f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "c363b27a245982df5369c0a19b890234"
server_key_iv: "17057f79b43e1c7f686a870dc96f66a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:07.302454 23279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:07.314551 23279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:07.318461 23279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:07.318666 23279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:07.319499 23279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:07.320679 23279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:07.320760 23279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.320835 23279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:07.320886 23279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.349231 23279 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:42649
I20260928 05:26:07.349279 23406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:42649 every 8 connection(s)
I20260928 05:26:07.350677 23279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:26:07.356312 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23279
I20260928 05:26:07.356909 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:44927
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=38895
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:26:07 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573167, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:07.370162 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.353016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:41735 (local address 127.8.124.254:45313)
0928 05:26:07.353174 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.353179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.354306 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 05:26:07.356054 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.356060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.356154 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.356269 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:07.357776 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.358731 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.360584 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.361170 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.361278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.364929 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:07.364934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:07.364938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:07.364969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:07.367302 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.367956 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.367960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.367962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.368080 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.368473 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.368478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.368480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.368758 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:07.368891 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.369556 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.369802 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:26:07.371150 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.353493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:41735)
0928 05:26:07.354146 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.354186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.355771 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.356469 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.356480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.356955 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:07.357557 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.357574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.358866 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.358869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.360433 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.360446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.361364 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.361369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.361637 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.362333 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:07.362357 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:07.364722 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:07.367428 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:07.367437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:07.367454 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:07.367841 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:07.368179 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:07.368184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:07.368188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:07.368366 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:07.368892 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:07.368900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:07.369205 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.369430 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.369969 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":241,"threads_started":1}
I20260928 05:26:07.372689 23407 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:07.372972 23407 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:07.373648 23407 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:07.375067 23090 ts_manager.cc:194] Registered new tserver with Master: f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649)
I20260928 05:26:07.376154 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:41735
W20260928 05:26:07.665001 23410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:07.665268 23410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:07.665369 23410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:07.672676 23410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:07.672797 23410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:07.672916 23410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:26:07.682709 23410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=38895
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:07.683935 23410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:07.685250 23410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:07.697527 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:07.697496 23418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:07.697476 23416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:07.698264 23410 server_base.cc:1034] running on GCE node
I20260928 05:26:07.698841 23410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:07.699680 23410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:07.700870 23410 hybrid_clock.cc:648] HybridClock initialized: now 1790573167700819 us; error 39 us; skew 500 ppm
I20260928 05:26:07.701139 23410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:07 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573167, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:07.703961 23410 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:26:07.705266 23410 webserver.cc:460] Webserver started at http://127.8.124.195:38895/ using document root <none> and password file <none>
I20260928 05:26:07.705971 23410 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:07.706106 23410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:07.711031 23410 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 05:26:07.713490 23425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.714777 23410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:07.714916 23410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "6ba075edfa804045986ea9100879a9fe"
format_stamp: "Formatted at 2026-09-28 05:26:05 on dist-test-slave-515x"
server_key: "ab26a0b13f291395d8a6dcc79acd7479"
server_key_iv: "153e6007c5888b0a1e2aca47e93539ec"
server_key_version: "encryptionkey@0"
I20260928 05:26:07.715842 23410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:07.734830 23410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:07.738809 23410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:07.739012 23410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:07.739835 23410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:07.741000 23410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:07.741091 23410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.741168 23410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:07.741217 23410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:07.767225 23410 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:44927
I20260928 05:26:07.767271 23537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:44927 every 8 connection(s)
I20260928 05:26:07.768633 23410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:26:07.777979 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23410
Sep 28 05:26:07 dist-test-slave-515x krb5kdc[22507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573167, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:07.787851 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.770932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:53027 (local address 127.8.124.254:45313)
0928 05:26:07.771105 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.771110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.772181 (+ 1071us) server_negotiation.cc:407] Connection header received
0928 05:26:07.773937 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.773942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.774021 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.774103 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:07.775475 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.776153 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.777624 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.778032 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.778154 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.782095 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:07.782099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:07.782103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:07.782131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:07.784924 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.785701 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.785704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.785706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.785865 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.786231 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.786235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.786243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.786590 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:07.786702 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.787460 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.787610 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:26:07.789156 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.771373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:53027)
0928 05:26:07.772034 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.772077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.773690 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.774332 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.774340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.774807 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:07.775323 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.775339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.776292 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.776296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.777475 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.777487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.778245 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.778253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.778476 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.779211 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:07.779246 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:07.781921 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:07.785044 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:07.785053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:07.785068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:07.785585 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:07.785961 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:07.785964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:07.785966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:07.786128 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:07.786708 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:07.786715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:07.787084 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.787379 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.787858 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":249,"threads_started":1}
I20260928 05:26:07.790557 23538 heartbeater.cc:347] Connected to a master server at 127.8.124.254:45313
I20260928 05:26:07.790822 23538 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:07.791472 23538 heartbeater.cc:511] Master 127.8.124.254:45313 requested a full tablet report, sending...
I20260928 05:26:07.792886 23090 ts_manager.cc:194] Registered new tserver with Master: 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927)
I20260928 05:26:07.793844 23090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:53027
I20260928 05:26:07.794724 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.778795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.8.124.254:45313)
0928 05:26:07.779388 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.779391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.781000 (+ 1609us) server_negotiation.cc:407] Connection header received
0928 05:26:07.781125 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.781131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.781194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.781273 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:07.783078 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.784191 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.785806 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.786352 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.786461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.789439 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:07.789444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:07.789447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:07.789473 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:07.792079 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.792765 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.792769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.792771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.792889 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:07.793278 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:07.793282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:07.793285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:07.793649 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:07.793866 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.794262 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.794431 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":514,"spinlock_wait_cycles":11136,"thread_start_us":385,"threads_started":1}
I20260928 05:26:07.797003 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:26:07.802750 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.797840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.8.124.254:45313)
0928 05:26:07.798011 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.798017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.798436 (+ 419us) server_negotiation.cc:407] Connection header received
0928 05:26:07.798536 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.798540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.798657 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.798760 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:07.799079 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.799732 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.801010 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.801410 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.801489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.801921 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:07.802348 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:26:07.802439 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.802490 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.802542 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:26:07.803498 23090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:26:07.810397 23090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:26:07.845682 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.821435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:44927 (local address 127.0.0.1:47048)
0928 05:26:07.836698 (+ 15263us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.837445 (+ 747us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.837680 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.838890 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.838897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.838935 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:07.839331 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.839343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.841116 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.841122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.843324 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.843336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.844606 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.844612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.844821 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.844884 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.845023 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.845214 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14938,"mutex_wait_us":1,"thread_start_us":15312,"threads_started":1}
I20260928 05:26:07.846517 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.821658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.8.124.195:44927)
0928 05:26:07.837996 (+ 16338us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.838006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.838042 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:26:07.838152 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.838160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.838456 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.839551 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:07.839787 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.841003 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.843465 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.844366 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.844523 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.844764 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.845077 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.845241 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16148,"thread_start_us":16927,"threads_started":1}
I20260928 05:26:07.847754 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.822360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52656)
0928 05:26:07.838855 (+ 16495us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.838870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.838952 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.840495 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.840500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.840526 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:07.840829 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.840839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.842893 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.842899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.844932 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.844943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.847052 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.847059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.847226 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.847257 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.847344 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.847463 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16378,"thread_start_us":15631,"threads_started":1}
I20260928 05:26:07.848775 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.821621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:42649 (local address 127.0.0.1:44108)
0928 05:26:07.838218 (+ 16597us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.838233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.838333 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.841742 (+ 3409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.841746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.841770 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:07.842213 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.842226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.844473 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.844479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.846455 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.846469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.848094 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.848100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.848265 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.848295 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.848384 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.848493 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16487,"thread_start_us":16974,"threads_started":1}
I20260928 05:26:07.848798 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.822227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.8.124.194:42649)
0928 05:26:07.841071 (+ 18844us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.841079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.841108 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:26:07.841195 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.841203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.841435 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.841636 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:07.842494 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.844341 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.846633 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.847913 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.848033 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.848192 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.848429 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.848542 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18695,"thread_start_us":18581,"threads_started":1}
I20260928 05:26:07.850078 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.822699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.8.124.193:41847)
0928 05:26:07.839866 (+ 17167us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.839876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.839909 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:26:07.840013 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.840023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.840282 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.840493 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:07.841021 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.842715 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.845098 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.846850 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.846982 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.847176 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.847387 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.847542 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17011,"thread_start_us":16873,"threads_started":1}
I20260928 05:26:07.850455 23473 tablet_service.cc:1467] Processing CreateTablet for tablet 06b2c831c9e44ff2880d23460ae5707f (DEFAULT_TABLE table=test-table [id=3353b09782e94bf0879a6b78a31d4f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:07.851754 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b2c831c9e44ff2880d23460ae5707f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:07.851894 23342 tablet_service.cc:1467] Processing CreateTablet for tablet 06b2c831c9e44ff2880d23460ae5707f (DEFAULT_TABLE table=test-table [id=3353b09782e94bf0879a6b78a31d4f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:07.853286 23342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b2c831c9e44ff2880d23460ae5707f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:07.853396 23210 tablet_service.cc:1467] Processing CreateTablet for tablet 06b2c831c9e44ff2880d23460ae5707f (DEFAULT_TABLE table=test-table [id=3353b09782e94bf0879a6b78a31d4f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:07.854655 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b2c831c9e44ff2880d23460ae5707f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:07.863445 23554 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Bootstrap starting.
I20260928 05:26:07.866916 23554 tablet_bootstrap.cc:654] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:07.869269 23554 log.cc:824] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:07.872515 23554 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: No bootstrap required, opened a new log
I20260928 05:26:07.872575 23556 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Bootstrap starting.
I20260928 05:26:07.872884 23554 ts_tablet_manager.cc:1397] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 05:26:07.876621 23556 tablet_bootstrap.cc:654] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:07.878022 23556 log.cc:824] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:07.883304 23554 raft_consensus.cc:348] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.883776 23554 raft_consensus.cc:374] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:07.883930 23554 raft_consensus.cc:729] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba075edfa804045986ea9100879a9fe, State: Initialized, Role: FOLLOWER
I20260928 05:26:07.884663 23554 consensus_queue.cc:260] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.887493 23556 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: No bootstrap required, opened a new log
I20260928 05:26:07.887830 23556 ts_tablet_manager.cc:1397] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260928 05:26:07.892238 23538 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:07.893333 23554 ts_tablet_manager.cc:1428] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260928 05:26:07.894177 23558 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Bootstrap starting.
I20260928 05:26:07.897943 23558 tablet_bootstrap.cc:654] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:07.899220 23559 raft_consensus.cc:482] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:26:07.899257 23558 log.cc:824] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:07.899488 23559 raft_consensus.cc:504] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.901851 23559 leader_election.cc:288] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649), 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:07.902056 23556 raft_consensus.cc:348] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.902439 23556 raft_consensus.cc:374] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:07.902582 23556 raft_consensus.cc:729] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f9b5b5bd07425ca2105cc673af192f, State: Initialized, Role: FOLLOWER
I20260928 05:26:07.903301 23556 consensus_queue.cc:260] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.909391 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.901471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:39663 (local address 127.8.124.194:42649)
0928 05:26:07.902066 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.902073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.902092 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:26:07.902156 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.902161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.902230 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.902323 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:07.902751 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.903775 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.906286 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.907609 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.907760 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.907848 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.909068 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.909191 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463}
I20260928 05:26:07.908730 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.901450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:42649 (local address 127.8.124.195:39663)
0928 05:26:07.901679 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.901702 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.901822 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.902372 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.902378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.902405 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:07.902588 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.902599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.903954 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.903961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.906084 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.906108 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.907939 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.907946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.908171 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.908208 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.908294 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.908441 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:26:07.910353 23558 tablet_bootstrap.cc:492] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: No bootstrap required, opened a new log
I20260928 05:26:07.910377 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b2c831c9e44ff2880d23460ae5707f" candidate_uuid: "6ba075edfa804045986ea9100879a9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9b5b5bd07425ca2105cc673af192f" is_pre_election: true
I20260928 05:26:07.910709 23558 ts_tablet_manager.cc:1397] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260928 05:26:07.910879 23362 raft_consensus.cc:2454] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ba075edfa804045986ea9100879a9fe in term 0.
I20260928 05:26:07.911710 23426 leader_election.cc:302] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ba075edfa804045986ea9100879a9fe, f4f9b5b5bd07425ca2105cc673af192f; no voters:
I20260928 05:26:07.912215 23559 raft_consensus.cc:2781] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:26:07.912448 23559 raft_consensus.cc:482] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:26:07.912714 23559 raft_consensus.cc:3037] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:07.916002 23559 raft_consensus.cc:504] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.917250 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.902630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:51609 (local address 127.8.124.193:41847)
0928 05:26:07.902810 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.902817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.909938 (+ 7121us) server_negotiation.cc:407] Connection header received
0928 05:26:07.910201 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.910206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.910290 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.910362 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:07.910935 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.912074 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.914559 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.915976 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.916110 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.916185 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.916804 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.916964 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:26:07.918140 23407 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:07.918798 23556 ts_tablet_manager.cc:1428] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Time spent starting tablet: real 0.031s user 0.009s sys 0.014s
I20260928 05:26:07.919658 23559 leader_election.cc:288] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 election: Requested vote from peers f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649), 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847)
I20260928 05:26:07.917851 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.902208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.8.124.195:51609)
0928 05:26:07.909864 (+ 7656us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:07.909879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:07.910029 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:07.910554 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:07.910559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:07.910585 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:07.910766 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.910775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.912261 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.912267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.914353 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:07.914368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.916383 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.916389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:07.916616 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.916651 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:26:07.917418 (+ 767us) client_negotiation.cc:240] Negotiation successful
0928 05:26:07.917568 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7561,"threads_started":1}
I20260928 05:26:07.919993 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b2c831c9e44ff2880d23460ae5707f" candidate_uuid: "6ba075edfa804045986ea9100879a9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f9b5b5bd07425ca2105cc673af192f"
I20260928 05:26:07.920205 23362 raft_consensus.cc:3037] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:07.921460 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b2c831c9e44ff2880d23460ae5707f" candidate_uuid: "6ba075edfa804045986ea9100879a9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f40593a597e43dc9b3a6e5086c9474f" is_pre_election: true
I20260928 05:26:07.922566 23229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b2c831c9e44ff2880d23460ae5707f" candidate_uuid: "6ba075edfa804045986ea9100879a9fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f40593a597e43dc9b3a6e5086c9474f"
W20260928 05:26:07.922847 23427 leader_election.cc:341] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847): Illegal state: must be running to vote when last-logged opid is not known
I20260928 05:26:07.923286 23362 raft_consensus.cc:2454] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ba075edfa804045986ea9100879a9fe in term 1.
W20260928 05:26:07.923362 23427 leader_election.cc:341] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847): Illegal state: must be running to vote when last-logged opid is not known
I20260928 05:26:07.924576 23558 raft_consensus.cc:348] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.924790 23426 leader_election.cc:302] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ba075edfa804045986ea9100879a9fe, f4f9b5b5bd07425ca2105cc673af192f; no voters: 4f40593a597e43dc9b3a6e5086c9474f
I20260928 05:26:07.925024 23558 raft_consensus.cc:374] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:07.925136 23558 raft_consensus.cc:729] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f40593a597e43dc9b3a6e5086c9474f, State: Initialized, Role: FOLLOWER
I20260928 05:26:07.925304 23559 raft_consensus.cc:2781] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:07.925980 23558 consensus_queue.cc:260] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.928828 23559 raft_consensus.cc:686] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 1 LEADER]: Becoming Leader. State: Replica: 6ba075edfa804045986ea9100879a9fe, State: Running, Role: LEADER
I20260928 05:26:07.929396 23559 consensus_queue.cc:237] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } }
I20260928 05:26:07.930297 23275 heartbeater.cc:503] Master 127.8.124.254:45313 was elected leader, sending a full tablet report...
I20260928 05:26:07.930941 23558 ts_tablet_manager.cc:1428] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s
I20260928 05:26:07.944103 23090 catalog_manager.cc:5495] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ba075edfa804045986ea9100879a9fe (127.8.124.195). New cstate: current_term: 1 leader_uuid: "6ba075edfa804045986ea9100879a9fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba075edfa804045986ea9100879a9fe" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 } health_report { overall_health: UNKNOWN } } }
I20260928 05:26:07.983094 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:07.977416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.8.124.195:44927)
0928 05:26:07.977576 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:07.977580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:07.977622 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:26:07.977718 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:07.977722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:07.977805 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:07.977921 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:07.978309 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.978997 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.981031 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:07.981424 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:07.981517 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:07.982095 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:07.982615 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:26:07.982718 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:07.982824 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 05:26:07.982886 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:26:07.990455 23229 raft_consensus.cc:3037] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:07.990832 23362 raft_consensus.cc:1261] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Refusing update from remote peer 6ba075edfa804045986ea9100879a9fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:07.991869 23563 consensus_queue.cc:1035] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f9b5b5bd07425ca2105cc673af192f" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:07.993032 23229 raft_consensus.cc:1261] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Refusing update from remote peer 6ba075edfa804045986ea9100879a9fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:07.993737 23563 consensus_queue.cc:1035] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f40593a597e43dc9b3a6e5086c9474f" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:08.013831 23568 mvcc.cc:205] Tried to move back new op lower bound from 7334187696080416768 to 7334187695849082880. Current Snapshot: MvccSnapshot[applied={T|T < 7334187696080416768}]
I20260928 05:26:08.022717 23090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:08.022931 23090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:08.026010 23090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Sending DeleteTablet for 3 replicas of tablet 06b2c831c9e44ff2880d23460ae5707f
I20260928 05:26:08.026973 23473 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b2c831c9e44ff2880d23460ae5707f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:08 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:47048
I20260928 05:26:08.027338 23342 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b2c831c9e44ff2880d23460ae5707f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:08 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:44108
I20260928 05:26:08.027603 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23127
I20260928 05:26:08.027626 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b2c831c9e44ff2880d23460ae5707f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:08 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:52656
I20260928 05:26:08.037748 23578 tablet_replica.cc:330] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: stopping tablet replica
I20260928 05:26:08.038275 23578 raft_consensus.cc:2229] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:08.042016 23578 raft_consensus.cc:2258] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:08.042907 23576 tablet_replica.cc:330] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: stopping tablet replica
I20260928 05:26:08.044380 23576 raft_consensus.cc:2229] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:26:08.048445 23576 raft_consensus.cc:2258] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:08.050300 23578 ts_tablet_manager.cc:1905] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:08.051667 23576 ts_tablet_manager.cc:1905] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:08.051239 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.028428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.8.124.193:41847)
0928 05:26:08.028598 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:08.028605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:08.028763 (+ 158us) server_negotiation.cc:407] Connection header received
0928 05:26:08.028932 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:08.028938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:08.029016 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:08.029117 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:08.030870 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.032103 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.034036 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.034711 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.034836 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:08.041434 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:08.041451 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:08.041471 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:08.041516 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:08.045950 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.046945 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.046954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.046958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.047107 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.047589 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.047594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.047597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.048549 (+ 952us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:08.048849 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:08.050603 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0928 05:26:08.050863 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:26:08.058562 23270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:08.062117 23577 tablet_replica.cc:330] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: stopping tablet replica
I20260928 05:26:08.062567 23577 raft_consensus.cc:2229] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:08.064426 23577 raft_consensus.cc:2258] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:08.065218 23576 ts_tablet_manager.cc:1918] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:08.065522 23576 log.cc:1196] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/06b2c831c9e44ff2880d23460ae5707f
I20260928 05:26:08.066280 23576 ts_tablet_manager.cc:1939] T 06b2c831c9e44ff2880d23460ae5707f P 6ba075edfa804045986ea9100879a9fe: Deleting consensus metadata
I20260928 05:26:08.066705 23577 ts_tablet_manager.cc:1905] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:08.068393 23077 catalog_manager.cc:4841] TS 6ba075edfa804045986ea9100879a9fe (127.8.124.195:44927): tablet 06b2c831c9e44ff2880d23460ae5707f (table test-table [id=3353b09782e94bf0879a6b78a31d4f9b]) successfully deleted
I20260928 05:26:08.073474 23577 ts_tablet_manager.cc:1918] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:08.073665 23577 log.cc:1196] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/06b2c831c9e44ff2880d23460ae5707f
I20260928 05:26:08.074198 23577 ts_tablet_manager.cc:1939] T 06b2c831c9e44ff2880d23460ae5707f P f4f9b5b5bd07425ca2105cc673af192f: Deleting consensus metadata
I20260928 05:26:08.076004 23074 catalog_manager.cc:4841] TS f4f9b5b5bd07425ca2105cc673af192f (127.8.124.194:42649): tablet 06b2c831c9e44ff2880d23460ae5707f (table test-table [id=3353b09782e94bf0879a6b78a31d4f9b]) successfully deleted
I20260928 05:26:08.154121 23578 ts_tablet_manager.cc:1918] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:08.154381 23578 log.cc:1196] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/06b2c831c9e44ff2880d23460ae5707f
I20260928 05:26:08.154852 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23127
W20260928 05:26:08.169461 23075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:41847 (error 108)
I20260928 05:26:08.170166 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23279
I20260928 05:26:08.171635 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.171134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52674)
0928 05:26:08.171376 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.171500 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 05:26:08.171851 23075 catalog_manager.cc:4584] TS 4f40593a597e43dc9b3a6e5086c9474f (127.8.124.193:41847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06b2c831c9e44ff2880d23460ae5707f: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
I20260928 05:26:08.185501 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.171041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.8.124.194:42649)
0928 05:26:08.171225 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:08.171231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:08.171787 (+ 556us) server_negotiation.cc:407] Connection header received
0928 05:26:08.171865 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:08.171870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:08.171961 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:08.172057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:08.173929 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.174722 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.176681 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.177074 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.177203 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:08.180379 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:08.180393 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:08.180408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:08.180440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:08.183037 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.183778 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.183785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.183787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.183891 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.184253 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.184255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.184257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.184583 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:08.184778 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:08.185085 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 05:26:08.185230 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:26:08.186223 23402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:08.225493 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.224969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52682)
0928 05:26:08.225188 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.225303 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:26:08.226621 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.226247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52698)
0928 05:26:08.226385 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.226470 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 05:26:08.272590 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23279
I20260928 05:26:08.286994 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.286546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52710)
0928 05:26:08.286742 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.286836 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 05:26:08.287988 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.287666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52712)
0928 05:26:08.287814 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.287881 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 05:26:08.293286 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23410
I20260928 05:26:08.310581 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.298024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.8.124.195:44927)
0928 05:26:08.298223 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:08.298229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:08.298249 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:26:08.298381 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:08.298387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:08.298456 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:08.298554 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:08.299811 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.300596 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.301934 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:08.302379 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:08.302491 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:08.304905 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:08.304916 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:08.304934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:08.304979 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:08.307897 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.308540 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.308550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.308554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.308711 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:08.309052 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:08.309057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:08.309061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:08.309392 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:08.309772 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:08.309978 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 05:26:08.310206 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:26:08.311205 23533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:08.368477 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.368046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52716)
0928 05:26:08.368238 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.368330 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:26:08.369539 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:08.369169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:41847 (local address 127.0.0.1:52722)
0928 05:26:08.369367 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:08.369436 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 05:26:08.430759 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23410
I20260928 05:26:08.446722 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23057
I20260928 05:26:08.447791 23120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:08.558667 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23057
I20260928 05:26:08.575758 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--webserver_interface=127.8.124.254
--webserver_port=42227
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:08.884922 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:08.885193 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:08.885294 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:08.892359 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:08.892453 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:08.892505 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:08.892565 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:08.892620 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:08.902535 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=42227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:08.903774 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:08.905051 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:08.915938 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:08.916019 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:08.915977 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:08.916851 23584 server_base.cc:1034] running on GCE node
I20260928 05:26:08.917654 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:08.919032 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:08.920225 23584 hybrid_clock.cc:648] HybridClock initialized: now 1790573168920173 us; error 40 us; skew 500 ppm
I20260928 05:26:08.920545 23584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:08 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:08.923440 23584 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:08.924819 23584 webserver.cc:460] Webserver started at http://127.8.124.254:42227/ using document root <none> and password file <none>
I20260928 05:26:08.925465 23584 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:08.925570 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:08.930744 23584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:26:08.933461 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:08.934790 23584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:08.934957 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "83b2a4438a1e4b258b2acff11f8e0653"
format_stamp: "Formatted at 2026-09-28 05:26:03 on dist-test-slave-515x"
server_key: "34422b46817bc13f997e122c8ed71476"
server_key_iv: "f19d4f12e112c05444e6bbec9ace01a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:08.935889 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:08.952879 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:08.956812 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:08.957036 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:08.973363 23584 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45313
I20260928 05:26:08.973382 23650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45313 every 8 connection(s)
I20260928 05:26:08.974743 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:08.981138 23651 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:26:08.984355 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23584
I20260928 05:26:08.984383 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap starting.
I20260928 05:26:08.985318 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:41847
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=44347
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:26:09.005782 23651 log.cc:824] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:09.021416 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:26:09.021972 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653: Bootstrap complete.
I20260928 05:26:09.030671 23651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:09.031519 23651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Initialized, Role: FOLLOWER
I20260928 05:26:09.032137 23651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:09.032341 23651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:09.032449 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:09.032605 23651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:26:09.036338 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:09.036793 23651 leader_election.cc:302] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b2a4438a1e4b258b2acff11f8e0653; no voters:
I20260928 05:26:09.037370 23651 leader_election.cc:288] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:26:09.037516 23655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:26:09.039391 23651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:09.041878 23655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [term 3 LEADER]: Becoming Leader. State: Replica: 83b2a4438a1e4b258b2acff11f8e0653, State: Running, Role: LEADER
I20260928 05:26:09.042546 23655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } }
I20260928 05:26:09.044859 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b2a4438a1e4b258b2acff11f8e0653. Latest consensus state: current_term: 3 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:09.045203 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:09.045717 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "83b2a4438a1e4b258b2acff11f8e0653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b2a4438a1e4b258b2acff11f8e0653" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45313 } } }
I20260928 05:26:09.045964 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b2a4438a1e4b258b2acff11f8e0653 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:09.054405 23663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:09.062345 23663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:09.063848 23663 catalog_manager.cc:1259] Loaded cluster ID: 5eaadf5a064347e7b9e4ef9480b6e5d7
I20260928 05:26:09.063998 23663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 05:26:09.066010 23663 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb7576bf9ad google::LogMessage::Fail() at ??:0
@ 0x7fb7576c288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb7576bf4de google::LogMessage::Flush() at ??:0
@ 0x7fb7576c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb76d0f2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb76d0ef976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb76d177610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb76d177439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb75a30b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb75a30e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb75a30e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb75a2e9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb75d5c36db start_thread at ??:0
@ 0x7fb75655961f clone at ??:0
W20260928 05:26:09.394459 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:09.394721 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:09.394819 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:09.401990 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:09.402088 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:09.402212 23653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:26:09.411577 23653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:41847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=44347
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:09.412742 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:09.413978 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:09.426126 23693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:09.426122 23696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:09.426100 23694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:09.426836 23653 server_base.cc:1034] running on GCE node
I20260928 05:26:09.427358 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:09.428215 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:09.429401 23653 hybrid_clock.cc:648] HybridClock initialized: now 1790573169429348 us; error 48 us; skew 500 ppm
I20260928 05:26:09.429698 23653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:09 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:09.432551 23653 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:26:09.433889 23653 webserver.cc:460] Webserver started at http://127.8.124.193:44347/ using document root <none> and password file <none>
I20260928 05:26:09.434525 23653 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:09.434621 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:09.439509 23653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 05:26:09.441974 23703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:09.443197 23653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 05:26:09.443327 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "4f40593a597e43dc9b3a6e5086c9474f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "8613dbd0efc2adac9982d11549f4dd19"
server_key_iv: "1bd371e9f736b2e418edb1674789aa6c"
server_key_version: "encryptionkey@0"
I20260928 05:26:09.444218 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:09.465701 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:09.469650 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:09.469867 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:09.470731 23653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:09.472364 23710 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 05:26:09.477368 23710 ts_tablet_manager.cc:1852] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 05:26:09.478374 23710 ts_tablet_manager.cc:1905] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:09.485137 23710 ts_tablet_manager.cc:1918] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:09.485359 23710 ts_tablet_manager.cc:1939] T 06b2c831c9e44ff2880d23460ae5707f P 4f40593a597e43dc9b3a6e5086c9474f: Deleting consensus metadata
I20260928 05:26:09.485828 23653 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 05:26:09.485932 23653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260928 05:26:09.486052 23653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:09.486121 23653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:09.515406 23653 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:41847
I20260928 05:26:09.515458 23816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:41847 every 8 connection(s)
I20260928 05:26:09.516683 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:26:09.519282 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23653
I20260928 05:26:09.519888 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:42649
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=34737
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:26:09.521873 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.519596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:36839)
0928 05:26:09.520404 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.520682 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":304,"threads_started":1}
W20260928 05:26:09.522461 23817 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:45313 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:45313: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
I20260928 05:26:09.523319 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.522949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:52607)
0928 05:26:09.523103 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.523186 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:26:09.524173 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.523820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:34257)
0928 05:26:09.523950 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.524035 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:26:09.524394 23817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:26:09.837672 23820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:09.837994 23820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:09.838153 23820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:09.845705 23820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:09.845813 23820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:09.845973 23820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:26:09.856057 23820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:42649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=34737
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:09.857338 23820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:09.858664 23820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:09.871063 23826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:09.871089 23828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:09.871244 23825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:09.872287 23820 server_base.cc:1034] running on GCE node
I20260928 05:26:09.872836 23820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:09.873719 23820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:09.874914 23820 hybrid_clock.cc:648] HybridClock initialized: now 1790573169874861 us; error 47 us; skew 500 ppm
I20260928 05:26:09.875212 23820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:09 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:09.878053 23820 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:26:09.879436 23820 webserver.cc:460] Webserver started at http://127.8.124.194:34737/ using document root <none> and password file <none>
I20260928 05:26:09.880060 23820 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:09.880156 23820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:09.885242 23820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:09.889520 23835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:09.890854 23820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:26:09.890987 23820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "f4f9b5b5bd07425ca2105cc673af192f"
format_stamp: "Formatted at 2026-09-28 05:26:04 on dist-test-slave-515x"
server_key: "c363b27a245982df5369c0a19b890234"
server_key_iv: "17057f79b43e1c7f686a870dc96f66a7"
server_key_version: "encryptionkey@0"
I20260928 05:26:09.891898 23820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:09.919603 23820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:09.923142 23820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:09.923352 23820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:09.924362 23820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:09.925645 23820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:09.925740 23820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:09.925820 23820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:09.925872 23820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:09.954499 23820 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:42649
I20260928 05:26:09.954569 23947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:42649 every 8 connection(s)
I20260928 05:26:09.956071 23820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:26:09.960343 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23820
I20260928 05:26:09.960807 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.958801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:36613)
0928 05:26:09.959501 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.959733 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260928 05:26:09.961059 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:44927
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:45313
--webserver_port=38895
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:09.961521 23948 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:45313 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:45313: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
I20260928 05:26:09.962394 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.962018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:58689)
0928 05:26:09.962174 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.962254 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:26:09.963201 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:09.962879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:53867)
0928 05:26:09.962998 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:09.963079 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 05:26:09.963446 23948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:26:10.267247 23951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:10.267503 23951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:10.267601 23951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:10.274658 23951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:10.274756 23951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:10.274871 23951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:26:10.284088 23951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=38895
--tserver_master_addrs=127.8.124.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:10.285244 23951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:10.286477 23951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:10.298575 23959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:10.298588 23957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:10.298550 23956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:10.299378 23951 server_base.cc:1034] running on GCE node
I20260928 05:26:10.299902 23951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:10.300772 23951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:10.301960 23951 hybrid_clock.cc:648] HybridClock initialized: now 1790573170301913 us; error 36 us; skew 500 ppm
I20260928 05:26:10.302243 23951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:10 dist-test-slave-515x krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573170, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:10.305007 23951 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:26:10.306324 23951 webserver.cc:460] Webserver started at http://127.8.124.195:38895/ using document root <none> and password file <none>
I20260928 05:26:10.306952 23951 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:10.307061 23951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:10.312054 23951 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:26:10.314576 23966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:10.315809 23951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:26:10.315945 23951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "6ba075edfa804045986ea9100879a9fe"
format_stamp: "Formatted at 2026-09-28 05:26:05 on dist-test-slave-515x"
server_key: "ab26a0b13f291395d8a6dcc79acd7479"
server_key_iv: "153e6007c5888b0a1e2aca47e93539ec"
server_key_version: "encryptionkey@0"
I20260928 05:26:10.316843 23951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:10.340430 23951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:10.344043 23951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:10.344295 23951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:10.345204 23951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:10.346539 23951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:10.346634 23951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:10.346738 23951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:10.346805 23951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:10.375590 23951 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:44927
I20260928 05:26:10.375646 24078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:44927 every 8 connection(s)
I20260928 05:26:10.376957 23951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:26:10.380070 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 23951
I20260928 05:26:10.381703 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:10.379702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:32775)
0928 05:26:10.380480 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:10.380780 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
W20260928 05:26:10.382259 24079 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:45313 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:45313: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
I20260928 05:26:10.382903 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:10.382634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:51589)
0928 05:26:10.382750 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:10.382810 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 05:26:10.383556 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:10.383300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:53499)
0928 05:26:10.383399 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:10.383462 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 05:26:10.383733 24079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:26:10.525560 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:10.524867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:51201)
0928 05:26:10.525262 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:10.525349 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":185,"threads_started":1}
I20260928 05:26:10.964628 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:10.963914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:46373)
0928 05:26:10.964328 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:10.964415 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":190,"threads_started":1}
I20260928 05:26:11.384860 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:11.384169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:40367)
0928 05:26:11.384563 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:11.384640 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":188,"threads_started":1}
I20260928 05:26:11.526975 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:11.526348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:46241)
0928 05:26:11.526734 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:11.526811 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":203,"threads_started":1}
I20260928 05:26:11.966132 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:11.965429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:41557)
0928 05:26:11.965890 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:11.965970 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260928 05:26:12.386305 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:12.385655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:44519)
0928 05:26:12.386080 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:12.386154 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260928 05:26:12.528672 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:12.527813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:48379)
0928 05:26:12.528388 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:12.528502 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":229,"threads_started":1}
I20260928 05:26:12.967535 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:12.966874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:50285)
0928 05:26:12.967290 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:12.967369 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":210,"threads_started":1}
I20260928 05:26:13.387656 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:13.387008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:44327)
0928 05:26:13.387414 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:13.387487 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":193,"threads_started":1}
I20260928 05:26:13.530179 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:13.529449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:54753)
0928 05:26:13.529913 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:13.529992 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260928 05:26:13.968987 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:13.968327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:39667)
0928 05:26:13.968723 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:13.968803 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":191,"threads_started":1}
I20260928 05:26:14.389067 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:14.388413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:46675)
0928 05:26:14.388795 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:14.388886 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":184,"threads_started":1}
I20260928 05:26:14.531567 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:14.530948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:58629)
0928 05:26:14.531348 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:14.531422 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
I20260928 05:26:14.970570 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:14.969896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:34521)
0928 05:26:14.970324 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:14.970405 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260928 05:26:15.390641 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:15.389930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:55889)
0928 05:26:15.390398 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:15.390478 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260928 05:26:15.533058 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:15.532374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:60745)
0928 05:26:15.532816 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:15.532893 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260928 05:26:15.972069 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:15.971380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:37175)
0928 05:26:15.971824 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:15.971911 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":221,"threads_started":1}
I20260928 05:26:16.392076 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:16.391418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:52239)
0928 05:26:16.391832 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:16.391918 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":201,"threads_started":1}
I20260928 05:26:16.534729 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:16.533974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:38957)
0928 05:26:16.534460 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:16.534540 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":233,"threads_started":1}
I20260928 05:26:16.973837 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:16.972946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:59685)
0928 05:26:16.973486 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:16.973581 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":236,"threads_started":1}
I20260928 05:26:17.393581 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:17.392861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:59407)
0928 05:26:17.393288 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:17.393382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":192,"threads_started":1}
I20260928 05:26:17.536201 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:17.535556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:58655)
0928 05:26:17.535948 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:17.536027 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260928 05:26:17.975395 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:17.974676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:60819)
0928 05:26:17.975134 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:17.975216 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260928 05:26:18.395028 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:18.394352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:43783)
0928 05:26:18.394786 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:18.394871 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":212,"threads_started":1}
I20260928 05:26:18.537683 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:18.536986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:49585)
0928 05:26:18.537411 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:18.537496 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":201,"threads_started":1}
I20260928 05:26:18.976953 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:18.976226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:55899)
0928 05:26:18.976672 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:18.976784 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260928 05:26:19.396600 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:19.395898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:47715)
0928 05:26:19.396355 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:19.396436 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":208,"threads_started":1}
I20260928 05:26:19.539141 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:19.538469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:56555)
0928 05:26:19.538892 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:19.538970 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260928 05:26:19.978525 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:19.977840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:57947)
0928 05:26:19.978283 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:19.978368 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":238,"threads_started":1}
I20260928 05:26:20.398175 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:20.397441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:49189)
0928 05:26:20.397928 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:20.398009 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":284,"threads_started":1}
I20260928 05:26:20.540624 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:20.539939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:48123)
0928 05:26:20.540375 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:20.540455 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260928 05:26:20.980083 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:20.979358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:49033)
0928 05:26:20.979817 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:20.979904 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":227,"threads_started":1}
I20260928 05:26:21.399852 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:21.398986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:33655)
0928 05:26:21.399489 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:21.399623 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260928 05:26:21.542326 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:21.541508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:59631)
0928 05:26:21.542003 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:21.542113 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
I20260928 05:26:21.981773 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:21.980994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:34819)
0928 05:26:21.981484 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:21.981571 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260928 05:26:22.401367 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:22.400656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:53635)
0928 05:26:22.401122 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:22.401209 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260928 05:26:22.544014 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:22.543274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:36603)
0928 05:26:22.543738 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:22.543844 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":229,"threads_started":1}
I20260928 05:26:22.983385 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:22.982601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:42733)
0928 05:26:22.983106 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:22.983195 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260928 05:26:23.403059 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:23.402246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:55957)
0928 05:26:23.402725 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:23.402824 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260928 05:26:23.545684 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:23.544915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:44129)
0928 05:26:23.545380 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:23.545464 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260928 05:26:23.985095 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:23.984352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:45369)
0928 05:26:23.984819 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:23.984914 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260928 05:26:24.404613 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:24.403954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:53219)
0928 05:26:24.404357 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:24.404438 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":200,"threads_started":1}
I20260928 05:26:24.547487 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:24.546782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.193:45845)
0928 05:26:24.547215 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:24.547328 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":199,"threads_started":1}
I20260928 05:26:24.986945 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:24.986241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.194:58787)
0928 05:26:24.986664 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:24.986753 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260928 05:26:25.389937 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23653
I20260928 05:26:25.406448 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:25.405670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:45313 (local address 127.8.124.195:55097)
0928 05:26:25.406166 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:25.406275 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
I20260928 05:26:25.406579 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:25.391031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.8.124.193:41847)
0928 05:26:25.391717 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:25.391736 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:25.391781 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:26:25.393362 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:25.393371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:25.393848 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:25.394136 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:25.395535 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.396747 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.397935 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.398579 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.398702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:25.401078 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:25.401085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:25.401099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:25.401138 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:25.403688 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.404285 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.404290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.404297 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.404437 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.404705 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.404708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.404710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.405177 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:25.405436 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:25.405886 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 05:26:25.406274 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":367,"threads_started":1}
I20260928 05:26:25.407948 23812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:25.491295 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23653
I20260928 05:26:25.504689 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23820
I20260928 05:26:25.520908 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:25.505649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.8.124.194:42649)
0928 05:26:25.506168 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:25.506182 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:25.506223 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:26:25.507738 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:25.507746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:25.508149 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:25.508434 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:25.509688 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.510897 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.512051 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.512689 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.512814 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:25.515259 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:25.515266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:25.515281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:25.515328 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:25.518096 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.518672 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.518677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.518683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.518836 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.519115 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.519117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.519119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.519604 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:25.519836 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:25.520153 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 05:26:25.520575 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
I20260928 05:26:25.522435 23943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:25.603539 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23820
I20260928 05:26:25.617368 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 23951
I20260928 05:26:25.634711 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:25.618362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.8.124.195:44927)
0928 05:26:25.618927 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:25.618941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:25.618979 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:26:25.620688 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:25.620701 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:25.621138 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:25.621480 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:25.623054 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.624255 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.625489 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:25.626185 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:25.626374 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:25.628876 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:25.628884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:25.628899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:25.628937 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:25.631504 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.632182 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.632189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.632195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.632364 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:25.632718 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:25.632722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:25.632724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:25.633210 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:25.633509 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:25.634033 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 05:26:25.634422 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":274,"threads_started":1}
I20260928 05:26:25.635999 24074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:25.717121 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 23951
2026-09-28T05:26:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23692 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:26:25 dist-test-slave-515x krb5kdc[24136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:26:25 dist-test-slave-515x krb5kdc[24136](info): set up 2 sockets
Sep 28 05:26:25 dist-test-slave-515x krb5kdc[24136](info): commencing operation
krb5kdc: starting...
W20260928 05:26:26.717835 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:26:26 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:26:26.931293 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:27.248942 24149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:27.249246 24149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:27.249385 24149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:27.256709 24149 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:26:27.256815 24149 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:26:27.256871 24149 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:26:27.256940 24149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:27.257015 24149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:27.257117 24149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:27.257180 24149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:27.257256 24149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:27.267096 24149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:43739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:27.268390 24149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:27.269707 24149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:27.280786 24155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:27.281172 24157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:27.281201 24154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:27.281692 24149 server_base.cc:1034] running on GCE node
I20260928 05:26:27.282459 24149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:27.283800 24149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:27.285034 24149 hybrid_clock.cc:648] HybridClock initialized: now 1790573187284972 us; error 32 us; skew 500 ppm
I20260928 05:26:27.285362 24149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:27 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573187, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:27.288301 24149 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:27.289664 24149 webserver.cc:460] Webserver started at http://127.8.124.254:39215/ using document root <none> and password file <none>
I20260928 05:26:27.290336 24149 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:27.290424 24149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:27.290719 24149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:27.293568 24149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "a9b416cf1f9c4223a51dc064d2675dc9"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "783ebdb31a9a431cd25f55cc4ba15cbd"
server_key_iv: "5c6091c3edccd405650e480cf0bfed03"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.294345 24149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "a9b416cf1f9c4223a51dc064d2675dc9"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "783ebdb31a9a431cd25f55cc4ba15cbd"
server_key_iv: "5c6091c3edccd405650e480cf0bfed03"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.299175 24149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:26:27.302469 24164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:27.303737 24149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:26:27.303892 24149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a9b416cf1f9c4223a51dc064d2675dc9"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "783ebdb31a9a431cd25f55cc4ba15cbd"
server_key_iv: "5c6091c3edccd405650e480cf0bfed03"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.304080 24149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:27.336476 24149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:27.340467 24149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:27.340713 24149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:27.357030 24149 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:43739
I20260928 05:26:27.357052 24215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:43739 every 8 connection(s)
I20260928 05:26:27.358376 24149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:27.361994 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24149
I20260928 05:26:27.362098 24216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:27.362193 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:26:27.362843 8691 external_mini_cluster.cc:1428] Setting key 5214979930b06936f8757fe6618b7697
I20260928 05:26:27.376034 24216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap starting.
Sep 28 05:26:27 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:27.379391 24216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:27.380434 24216 log.cc:824] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:27.383045 24216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: No bootstrap required, opened a new log
I20260928 05:26:27.385169 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:27.364717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.8.124.254:43739)
0928 05:26:27.365477 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:27.365491 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:27.365541 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:26:27.367119 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:27.367146 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:27.367695 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:27.368134 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:27.369410 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.371283 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.372738 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.373822 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.374000 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:27.377798 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:27.377809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:27.377827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:27.377871 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:27.381029 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:27.381708 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:27.381714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:27.381720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:27.381886 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:27.382251 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:27.382254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:27.382256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:27.382735 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:27.383019 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:27.383499 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 05:26:27.383917 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":314,"threads_started":1}
I20260928 05:26:27.391330 24216 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:27.391637 24216 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:27.391706 24216 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Initialized, Role: FOLLOWER
I20260928 05:26:27.392160 24216 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:27.392354 24216 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:27.392443 24216 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:27.392556 24216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:27.394644 24216 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:27.395063 24216 leader_election.cc:302] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9b416cf1f9c4223a51dc064d2675dc9; no voters:
I20260928 05:26:27.395540 24216 leader_election.cc:288] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:26:27.395665 24221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:27.396258 24221 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 LEADER]: Becoming Leader. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Running, Role: LEADER
I20260928 05:26:27.396750 24221 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:27.397380 24216 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:27.399080 24222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:26:27.399386 24222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:27.400260 24227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:27.400025 24223 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9b416cf1f9c4223a51dc064d2675dc9. Latest consensus state: current_term: 1 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:26:27.401988 24223 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:27.406191 24227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:27.415637 24227 catalog_manager.cc:1347] Generated new cluster ID: 81c4b66ee3734532a8a2680bb1b69ede
I20260928 05:26:27.415740 24227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:26:27.424255 24227 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:26:27.425453 24227 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:26:27.435719 24227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Generated new TSK 0
I20260928 05:26:27.436472 24227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:26:27.496953 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:27.813206 24242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:27.813472 24242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:27.813640 24242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:27.820863 24242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:27.820966 24242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:27.821116 24242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:26:27.830752 24242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:27.832031 24242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:27.833298 24242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:27.845582 24250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:27.845687 24248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:27.845680 24247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:27.846501 24242 server_base.cc:1034] running on GCE node
I20260928 05:26:27.847056 24242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:27.847949 24242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:27.849145 24242 hybrid_clock.cc:648] HybridClock initialized: now 1790573187849090 us; error 48 us; skew 500 ppm
I20260928 05:26:27.849442 24242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:27 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573187, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:27.852411 24242 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:26:27.853799 24242 webserver.cc:460] Webserver started at http://127.8.124.193:40077/ using document root <none> and password file <none>
I20260928 05:26:27.854467 24242 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:27.854583 24242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:27.854879 24242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:27.857795 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "26a2c5ead44b4c17837e9e1452d2836b"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "fcf4aaf9620e05fff699cc1330fed9e8"
server_key_iv: "e2f11e22b07593af365e3811d9023d8f"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.858536 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "26a2c5ead44b4c17837e9e1452d2836b"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "fcf4aaf9620e05fff699cc1330fed9e8"
server_key_iv: "e2f11e22b07593af365e3811d9023d8f"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.863384 24242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:26:27.866614 24257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:27.867892 24242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:27.868039 24242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "26a2c5ead44b4c17837e9e1452d2836b"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "fcf4aaf9620e05fff699cc1330fed9e8"
server_key_iv: "e2f11e22b07593af365e3811d9023d8f"
server_key_version: "encryptionkey@0"
I20260928 05:26:27.868206 24242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:27.888680 24242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:27.893422 24242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:27.893729 24242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:27.894630 24242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:27.895869 24242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:27.895962 24242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:27.896080 24242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:27.896129 24242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:27.926359 24242 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:35613
I20260928 05:26:27.926393 24369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:35613 every 8 connection(s)
I20260928 05:26:27.927574 24242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:26:27.928234 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24242
I20260928 05:26:27.928453 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:26:27.929169 8691 external_mini_cluster.cc:1428] Setting key d6de80d348242fd5dcb3e6391ad4f3c2
Sep 28 05:26:27 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573187, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:27.947407 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:27.930139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:37737 (local address 127.8.124.254:43739)
0928 05:26:27.930529 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:27.930533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:27.931640 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 05:26:27.933435 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:27.933440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:27.933514 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:27.933664 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:27.935352 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.936185 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.937684 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.938083 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.938191 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:27.941876 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:27.941883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:27.941888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:27.941930 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:27.944331 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:27.945108 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:27.945114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:27.945118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:27.945236 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:27.945721 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:27.945726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:27.945728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:27.946005 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:27.946139 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:27.946941 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 05:26:27.947163 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":187,"threads_started":1}
I20260928 05:26:27.948709 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:27.930704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:37737)
0928 05:26:27.931487 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:27.931546 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:27.933167 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:27.933936 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:27.933944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:27.934484 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:27.935133 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:27.935153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.936324 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.936329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:27.937527 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:27.937539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:27.938300 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:27.938308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:27.938513 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:27.939141 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:27.939173 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:27.941640 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:27.944462 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:27.944470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:27.944485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:27.944987 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:27.945332 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:27.945337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:27.945340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:27.945560 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:27.946124 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:27.946131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:27.946536 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 05:26:27.946822 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 05:26:27.947331 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":280,"threads_started":1}
I20260928 05:26:27.950402 24370 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:27.950698 24370 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:27.951397 24370 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
I20260928 05:26:27.953454 24181 ts_manager.cc:194] Registered new tserver with Master: 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
I20260928 05:26:27.954968 24181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:37737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:26:27.980988 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:28.296365 24376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:28.296618 24376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:28.296737 24376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:28.303977 24376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:28.304075 24376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:28.304220 24376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:26:28.313632 24376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:28.314785 24376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:28.315987 24376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:28.328414 24381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:28.328413 24384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:28.328442 24382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:28.329243 24376 server_base.cc:1034] running on GCE node
I20260928 05:26:28.329839 24376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:28.330734 24376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:28.331919 24376 hybrid_clock.cc:648] HybridClock initialized: now 1790573188331865 us; error 46 us; skew 500 ppm
I20260928 05:26:28.332237 24376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:28 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573188, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:28.337734 24376 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:26:28.339094 24376 webserver.cc:460] Webserver started at http://127.8.124.194:41515/ using document root <none> and password file <none>
I20260928 05:26:28.339697 24376 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:28.339793 24376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:28.340039 24376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:28.342801 24376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "af3e8642856b4aa79a25c7488b63f720"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "04d5066e61a98500d2db90deea8a0d4f"
server_key_iv: "699bf68b501c48b58ed9fd9d044617f4"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.343463 24376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "af3e8642856b4aa79a25c7488b63f720"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "04d5066e61a98500d2db90deea8a0d4f"
server_key_iv: "699bf68b501c48b58ed9fd9d044617f4"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.348230 24376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 05:26:28.351441 24391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.352651 24376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:26:28.352787 24376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "af3e8642856b4aa79a25c7488b63f720"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "04d5066e61a98500d2db90deea8a0d4f"
server_key_iv: "699bf68b501c48b58ed9fd9d044617f4"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.352946 24376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:28.371040 24376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:28.375774 24376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:28.376035 24376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:28.377065 24376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:28.378463 24376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:28.378530 24376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.378623 24376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:28.378664 24376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.406527 24376 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:38247
I20260928 05:26:28.406589 24503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:38247 every 8 connection(s)
I20260928 05:26:28.407790 24376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:26:28.412380 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24376
I20260928 05:26:28.412546 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:26:28.412950 8691 external_mini_cluster.cc:1428] Setting key 2eff2c444b83af2af8f1baf4c0a02765
Sep 28 05:26:28 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573188, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:28.427428 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.410167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:38301 (local address 127.8.124.254:43739)
0928 05:26:28.410364 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.410370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.411471 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 05:26:28.413283 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.413289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.413371 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.413482 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:28.415427 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.416419 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.417999 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.418540 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.418660 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.422002 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:28.422009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:28.422014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:28.422052 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:28.424548 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.425321 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.425324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.425326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.425441 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.425879 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.425882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.425884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.426156 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:28.426267 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.427002 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.427175 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:26:28.428833 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.410649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:38301)
0928 05:26:28.411317 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:28.411358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:28.413006 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:28.413955 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:28.413967 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:28.414628 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:28.415262 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.415279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.416558 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.416562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.417872 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.417883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.418725 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.418729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.418934 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.419566 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:28.419586 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:28.421775 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:28.424674 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:28.424682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:28.424699 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:28.425210 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:28.425534 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:28.425539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:28.425542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:28.425781 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:28.426260 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:28.426266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:28.426650 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 05:26:28.426940 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 05:26:28.427438 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260928 05:26:28.430362 24504 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:28.430657 24504 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:28.431375 24504 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
I20260928 05:26:28.432832 24181 ts_manager.cc:194] Registered new tserver with Master: af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247)
I20260928 05:26:28.433591 24181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:38301
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:26:28.464354 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:28.775928 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:28.776225 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:28.776364 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:28.783839 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:28.784055 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:28.784248 24509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:26:28.794009 24509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:28.795276 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:28.796548 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:28.809062 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:28.809121 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:28.809504 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:28.809899 24509 server_base.cc:1034] running on GCE node
I20260928 05:26:28.810467 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:28.811373 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:28.812558 24509 hybrid_clock.cc:648] HybridClock initialized: now 1790573188812511 us; error 38 us; skew 500 ppm
I20260928 05:26:28.812837 24509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:28 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573188, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:28.815795 24509 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:26:28.817181 24509 webserver.cc:460] Webserver started at http://127.8.124.195:45201/ using document root <none> and password file <none>
I20260928 05:26:28.817850 24509 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:28.817963 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:28.818248 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:26:28.821002 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "e001b01d363dd01098dea4dcd5dcf05b"
server_key_iv: "835db6ab9450458cab87457ca5a28f85"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.821754 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "e001b01d363dd01098dea4dcd5dcf05b"
server_key_iv: "835db6ab9450458cab87457ca5a28f85"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.826596 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 05:26:28.829787 24524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.831003 24509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:26:28.831144 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "e001b01d363dd01098dea4dcd5dcf05b"
server_key_iv: "835db6ab9450458cab87457ca5a28f85"
server_key_version: "encryptionkey@0"
I20260928 05:26:28.831324 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:28.853524 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:28.857312 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:28.857561 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:28.858453 24509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:28.859727 24509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:28.859807 24509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.859922 24509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:28.859973 24509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:28.888306 24509 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:36913
I20260928 05:26:28.888366 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:36913 every 8 connection(s)
I20260928 05:26:28.889492 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:26:28.894611 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24509
I20260928 05:26:28.894781 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:26:28.895337 8691 external_mini_cluster.cc:1428] Setting key ca2b9a371c17fa3ab2f48ef6fff6da71
Sep 28 05:26:28 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573188, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:28.908851 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.891897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:55205 (local address 127.8.124.254:43739)
0928 05:26:28.892090 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.892097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.893592 (+ 1495us) server_negotiation.cc:407] Connection header received
0928 05:26:28.895966 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.895970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.896040 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.896120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:28.897782 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.898677 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.900166 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.900570 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.900673 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.903885 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:28.903890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:28.903894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:28.903926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:28.906339 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.906907 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.906910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.906912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.907054 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.907397 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.907401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.907403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.907646 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:28.907749 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.908364 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.908573 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:26:28.909869 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.892502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:55205)
0928 05:26:28.893399 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:28.893468 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:28.895682 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:28.896370 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:28.896382 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:28.896933 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:28.897557 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.897576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.898844 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.898848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.900028 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.900040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.900767 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.900783 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.900995 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.901584 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:28.901635 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:28.903718 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:28.906428 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:28.906433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:28.906443 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:28.906822 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:28.907130 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:28.907133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:28.907136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:28.907300 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:28.907730 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:28.907735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:28.908015 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 05:26:28.908263 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 05:26:28.908604 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":332,"threads_started":1}
I20260928 05:26:28.911039 24637 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:28.911309 24637 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:28.911942 24637 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
I20260928 05:26:28.913123 24181 ts_manager.cc:194] Registered new tserver with Master: d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913)
I20260928 05:26:28.913913 24181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:55205
I20260928 05:26:28.922434 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:26:28.938581 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.926662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.8.124.254:43739)
0928 05:26:28.926872 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.926877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.927200 (+ 323us) server_negotiation.cc:407] Connection header received
0928 05:26:28.927337 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.927341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.927413 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.927512 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:28.928828 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.929968 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.931428 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.931841 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.931936 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.934092 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:28.934097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:28.934100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:28.934134 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:28.936484 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.937038 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.937041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.937044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.937159 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:28.937453 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:28.937456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:28.937458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:28.937804 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:28.937982 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.938206 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.938362 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:26:28.942203 24181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:26:28.948858 24181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:26:28.957962 24370 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:28.987661 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.962468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.8.124.195:36913)
0928 05:26:28.978693 (+ 16225us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.978702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.979467 (+ 765us) server_negotiation.cc:407] Connection header received
0928 05:26:28.979590 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.979600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.979918 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.980113 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:28.980817 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.982379 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.985198 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.986552 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.986666 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.986924 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.987168 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.987335 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16047,"thread_start_us":15896,"threads_started":1}
I20260928 05:26:28.987797 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.961807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36913 (local address 127.0.0.1:60626)
0928 05:26:28.979160 (+ 17353us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:28.979181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:28.979333 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:28.980152 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:28.980160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:28.980193 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:28.980608 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.980626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.982532 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.982537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.984991 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.985005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.986726 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.986732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.986913 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.986964 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 05:26:28.987108 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:26:28.987302 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17207,"mutex_wait_us":60,"thread_start_us":11137,"threads_started":1}
I20260928 05:26:28.988094 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.961260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:35613 (local address 127.0.0.1:59462)
0928 05:26:28.973338 (+ 12078us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:28.973392 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:28.973594 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:28.981360 (+ 7766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:28.981365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:28.981389 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:28.982033 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.982044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.983925 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.983930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.985460 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.985474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.987263 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.987269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.987435 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.987468 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:26:28.987535 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 05:26:28.987663 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11795,"mutex_wait_us":3,"thread_start_us":18511,"threads_started":1}
I20260928 05:26:28.988679 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.962840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.8.124.194:38247)
0928 05:26:28.978187 (+ 15347us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.978199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.980375 (+ 2176us) server_negotiation.cc:407] Connection header received
0928 05:26:28.980494 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.980503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.980740 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.980938 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:28.981471 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.982857 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.985147 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.986596 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.986770 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.986952 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.987912 (+ 960us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.988394 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15150,"thread_start_us":14992,"threads_started":1}
I20260928 05:26:28.987818 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.961921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38247 (local address 127.0.0.1:49702)
0928 05:26:28.980071 (+ 18150us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:28.980090 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:28.980230 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:28.980922 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:28.980926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:28.980951 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:28.981270 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.981293 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.982982 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.982988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.984995 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:28.985005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.987069 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.987075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:28.987222 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.987256 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:26:28.987332 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 05:26:28.987442 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18006,"mutex_wait_us":5,"thread_start_us":16941,"threads_started":1}
I20260928 05:26:28.989189 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:28.961399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.8.124.193:35613)
0928 05:26:28.980821 (+ 19422us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:28.980828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:28.980850 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:26:28.980934 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:28.980941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:28.981160 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:28.981352 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:28.982220 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.983746 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.985666 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:28.987027 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:28.988527 (+ 1500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:28.988677 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:28.988734 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 05:26:28.988883 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19251,"thread_start_us":19103,"threads_started":1}
I20260928 05:26:28.991788 24572 tablet_service.cc:1467] Processing CreateTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 (DEFAULT_TABLE table=test-table [id=954430245b4d4f01944d7e0ec62f170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:28.992978 24572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8beee5f9124a7a8b3baf1c5526ee65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:28.992466 24439 tablet_service.cc:1467] Processing CreateTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 (DEFAULT_TABLE table=test-table [id=954430245b4d4f01944d7e0ec62f170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:28.992466 24305 tablet_service.cc:1467] Processing CreateTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 (DEFAULT_TABLE table=test-table [id=954430245b4d4f01944d7e0ec62f170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:28.993804 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8beee5f9124a7a8b3baf1c5526ee65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:28.993806 24439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8beee5f9124a7a8b3baf1c5526ee65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:29.004730 24655 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Bootstrap starting.
I20260928 05:26:29.007757 24657 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Bootstrap starting.
I20260928 05:26:29.008409 24656 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Bootstrap starting.
I20260928 05:26:29.008639 24655 tablet_bootstrap.cc:654] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:29.009950 24655 log.cc:824] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:29.011433 24657 tablet_bootstrap.cc:654] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:29.011839 24656 tablet_bootstrap.cc:654] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:29.012346 24657 log.cc:824] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:29.013072 24656 log.cc:824] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:29.019217 24655 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: No bootstrap required, opened a new log
I20260928 05:26:29.019493 24655 ts_tablet_manager.cc:1397] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260928 05:26:29.020107 24656 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: No bootstrap required, opened a new log
I20260928 05:26:29.020318 24657 tablet_bootstrap.cc:492] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: No bootstrap required, opened a new log
I20260928 05:26:29.020329 24656 ts_tablet_manager.cc:1397] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260928 05:26:29.020566 24657 ts_tablet_manager.cc:1397] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260928 05:26:29.027065 24656 raft_consensus.cc:348] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.027595 24656 raft_consensus.cc:374] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:29.027760 24656 raft_consensus.cc:729] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a2c5ead44b4c17837e9e1452d2836b, State: Initialized, Role: FOLLOWER
I20260928 05:26:29.028779 24656 consensus_queue.cc:260] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.029804 24655 raft_consensus.cc:348] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.030305 24655 raft_consensus.cc:374] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:29.030450 24655 raft_consensus.cc:729] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f5e2b6a77e4aebad6055203c8cf310, State: Initialized, Role: FOLLOWER
I20260928 05:26:29.030519 24656 ts_tablet_manager.cc:1428] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260928 05:26:29.030447 24657 raft_consensus.cc:348] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.030897 24657 raft_consensus.cc:374] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:29.031003 24657 raft_consensus.cc:729] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3e8642856b4aa79a25c7488b63f720, State: Initialized, Role: FOLLOWER
I20260928 05:26:29.031234 24655 consensus_queue.cc:260] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.031535 24657 consensus_queue.cc:260] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.032311 24637 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:29.034101 24655 ts_tablet_manager.cc:1428] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 05:26:29.034974 24504 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:29.036288 24657 ts_tablet_manager.cc:1428] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260928 05:26:29.140980 24638 tablet.cc:2369] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:26:29.159688 24505 tablet.cc:2369] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:26:29.179919 24371 tablet.cc:2369] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:26:29.230439 24661 raft_consensus.cc:482] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:26:29.230715 24661 raft_consensus.cc:504] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.232323 24661 leader_election.cc:288] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247), d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913)
I20260928 05:26:29.238783 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.232794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:56799 (local address 127.8.124.194:38247)
0928 05:26:29.232996 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.233003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.233772 (+ 769us) server_negotiation.cc:407] Connection header received
0928 05:26:29.233953 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.233957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.234026 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.234107 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:29.234687 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.235432 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.237166 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.237926 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.238068 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.238139 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.238417 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.238535 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:26:29.238915 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.232832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38247 (local address 127.8.124.193:56799)
0928 05:26:29.233670 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:29.233685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:29.233808 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:29.234126 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:29.234130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:29.234152 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:29.234549 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:29.234563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.235570 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.235574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:29.237018 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:29.237029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.238112 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.238118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:29.238262 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.238298 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:26:29.238375 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:26:29.238497 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":43,"thread_start_us":410,"threads_started":1}
I20260928 05:26:29.238948 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.232852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:41761 (local address 127.8.124.195:36913)
0928 05:26:29.233059 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.233065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.233865 (+ 800us) server_negotiation.cc:407] Connection header received
0928 05:26:29.234097 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.234102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.234162 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.234259 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:29.234842 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.235668 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.237332 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.238123 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.238261 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.238329 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.238584 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.238702 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:26:29.239069 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.232865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36913 (local address 127.8.124.193:41761)
0928 05:26:29.233764 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:29.233781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:29.233919 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:29.234279 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:29.234283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:29.234317 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:29.234688 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:29.234697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.235797 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.235803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:29.237188 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:29.237199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.238322 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.238328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:29.238454 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.238475 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:26:29.238550 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:26:29.238668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":1,"thread_start_us":397,"threads_started":1}
I20260928 05:26:29.240077 24592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8beee5f9124a7a8b3baf1c5526ee65" candidate_uuid: "26a2c5ead44b4c17837e9e1452d2836b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" is_pre_election: true
I20260928 05:26:29.240077 24459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8beee5f9124a7a8b3baf1c5526ee65" candidate_uuid: "26a2c5ead44b4c17837e9e1452d2836b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3e8642856b4aa79a25c7488b63f720" is_pre_election: true
I20260928 05:26:29.240653 24592 raft_consensus.cc:2454] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a2c5ead44b4c17837e9e1452d2836b in term 0.
I20260928 05:26:29.240653 24459 raft_consensus.cc:2454] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a2c5ead44b4c17837e9e1452d2836b in term 0.
I20260928 05:26:29.241425 24260 leader_election.cc:302] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a2c5ead44b4c17837e9e1452d2836b, d0f5e2b6a77e4aebad6055203c8cf310; no voters:
I20260928 05:26:29.241951 24661 raft_consensus.cc:2781] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:26:29.242095 24661 raft_consensus.cc:482] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:26:29.242204 24661 raft_consensus.cc:3037] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:29.244248 24661 raft_consensus.cc:504] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.244798 24661 leader_election.cc:288] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [CANDIDATE]: Term 1 election: Requested vote from peers af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247), d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913)
I20260928 05:26:29.245235 24592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8beee5f9124a7a8b3baf1c5526ee65" candidate_uuid: "26a2c5ead44b4c17837e9e1452d2836b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
I20260928 05:26:29.245211 24459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8beee5f9124a7a8b3baf1c5526ee65" candidate_uuid: "26a2c5ead44b4c17837e9e1452d2836b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3e8642856b4aa79a25c7488b63f720"
I20260928 05:26:29.245419 24592 raft_consensus.cc:3037] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:29.245419 24459 raft_consensus.cc:3037] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:29.248363 24592 raft_consensus.cc:2454] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a2c5ead44b4c17837e9e1452d2836b in term 1.
I20260928 05:26:29.248363 24459 raft_consensus.cc:2454] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a2c5ead44b4c17837e9e1452d2836b in term 1.
I20260928 05:26:29.248708 24259 leader_election.cc:302] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a2c5ead44b4c17837e9e1452d2836b, af3e8642856b4aa79a25c7488b63f720; no voters:
I20260928 05:26:29.248963 24661 raft_consensus.cc:2781] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:29.249349 24661 raft_consensus.cc:686] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 1 LEADER]: Becoming Leader. State: Replica: 26a2c5ead44b4c17837e9e1452d2836b, State: Running, Role: LEADER
I20260928 05:26:29.249846 24661 consensus_queue.cc:237] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } }
I20260928 05:26:29.253934 24181 catalog_manager.cc:5495] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b reported cstate change: term changed from 0 to 1, leader changed from <none> to 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193). New cstate: current_term: 1 leader_uuid: "26a2c5ead44b4c17837e9e1452d2836b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } health_report { overall_health: UNKNOWN } } }
I20260928 05:26:29.281242 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.275973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.8.124.193:35613)
0928 05:26:29.276159 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.276165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.276234 (+ 69us) server_negotiation.cc:407] Connection header received
0928 05:26:29.276471 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.276475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.276546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.276666 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:29.277045 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.277784 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.279211 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.279614 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.279718 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.280204 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:29.280688 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:26:29.280815 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.280920 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.280996 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:26:29.288868 24459 raft_consensus.cc:1261] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Refusing update from remote peer 26a2c5ead44b4c17837e9e1452d2836b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:29.288868 24592 raft_consensus.cc:1261] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Refusing update from remote peer 26a2c5ead44b4c17837e9e1452d2836b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:29.289868 24666 consensus_queue.cc:1035] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:29.290216 24661 consensus_queue.cc:1035] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:29.301990 24669 mvcc.cc:205] Tried to move back new op lower bound from 7334187783317696512 to 7334187783170809856. Current Snapshot: MvccSnapshot[applied={T|T < 7334187783317696512}]
I20260928 05:26:29.302243 24670 mvcc.cc:205] Tried to move back new op lower bound from 7334187783317696512 to 7334187783170809856. Current Snapshot: MvccSnapshot[applied={T|T < 7334187783317696512}]
I20260928 05:26:29.302614 24671 mvcc.cc:205] Tried to move back new op lower bound from 7334187783317696512 to 7334187783170809856. Current Snapshot: MvccSnapshot[applied={T|T < 7334187783317696512}]
I20260928 05:26:29.311578 24181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:29.311868 24181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:29.315251 24181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Sending DeleteTablet for 3 replicas of tablet db8beee5f9124a7a8b3baf1c5526ee65
I20260928 05:26:29.316222 24305 tablet_service.cc:1514] Processing DeleteTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:29 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:59462
I20260928 05:26:29.316264 24439 tablet_service.cc:1514] Processing DeleteTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:29 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:49702
I20260928 05:26:29.316507 24572 tablet_service.cc:1514] Processing DeleteTablet for tablet db8beee5f9124a7a8b3baf1c5526ee65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:29 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:60626
I20260928 05:26:29.316917 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24242
I20260928 05:26:29.328626 24679 tablet_replica.cc:330] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: stopping tablet replica
I20260928 05:26:29.329617 24678 tablet_replica.cc:330] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: stopping tablet replica
I20260928 05:26:29.329905 24680 tablet_replica.cc:330] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: stopping tablet replica
I20260928 05:26:29.330137 24678 raft_consensus.cc:2229] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:26:29.330632 24679 raft_consensus.cc:2229] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:29.330842 24678 raft_consensus.cc:2258] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:29.331017 24680 raft_consensus.cc:2229] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:29.332069 24680 raft_consensus.cc:2258] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:29.332638 24679 raft_consensus.cc:2258] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:29.334131 24678 ts_tablet_manager.cc:1905] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:29.334743 24680 ts_tablet_manager.cc:1905] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:26:29 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:26:29.341205 24679 ts_tablet_manager.cc:1905] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:29.341240 24680 ts_tablet_manager.cc:1918] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:29.341379 24680 log.cc:1196] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/db8beee5f9124a7a8b3baf1c5526ee65
I20260928 05:26:29.341843 24680 ts_tablet_manager.cc:1939] T db8beee5f9124a7a8b3baf1c5526ee65 P d0f5e2b6a77e4aebad6055203c8cf310: Deleting consensus metadata
I20260928 05:26:29.343642 24167 catalog_manager.cc:4841] TS d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913): tablet db8beee5f9124a7a8b3baf1c5526ee65 (table test-table [id=954430245b4d4f01944d7e0ec62f170b]) successfully deleted
I20260928 05:26:29.343633 24678 ts_tablet_manager.cc:1918] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:29.344074 24678 log.cc:1196] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/db8beee5f9124a7a8b3baf1c5526ee65
I20260928 05:26:29.344417 24678 ts_tablet_manager.cc:1939] T db8beee5f9124a7a8b3baf1c5526ee65 P 26a2c5ead44b4c17837e9e1452d2836b: Deleting consensus metadata
I20260928 05:26:29.345881 24168 catalog_manager.cc:4841] TS 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613): tablet db8beee5f9124a7a8b3baf1c5526ee65 (table test-table [id=954430245b4d4f01944d7e0ec62f170b]) successfully deleted
I20260928 05:26:29.347240 24679 ts_tablet_manager.cc:1918] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:29.347404 24679 log.cc:1196] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/db8beee5f9124a7a8b3baf1c5526ee65
I20260928 05:26:29.347864 24679 ts_tablet_manager.cc:1939] T db8beee5f9124a7a8b3baf1c5526ee65 P af3e8642856b4aa79a25c7488b63f720: Deleting consensus metadata
I20260928 05:26:29.349551 24166 catalog_manager.cc:4841] TS af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247): tablet db8beee5f9124a7a8b3baf1c5526ee65 (table test-table [id=954430245b4d4f01944d7e0ec62f170b]) successfully deleted
I20260928 05:26:29.350476 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.318332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.8.124.193:35613)
0928 05:26:29.318509 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.318515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.328658 (+ 10143us) server_negotiation.cc:407] Connection header received
0928 05:26:29.329176 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.329182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.329312 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.329439 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:29.331458 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.332515 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.334663 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.336076 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.336182 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.341875 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:29.341888 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:29.341905 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:29.341949 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:29.347190 (+ 5241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.348139 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.348148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.348152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.348325 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.348840 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.348845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.348848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.349260 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:29.349531 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.349938 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.350208 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:26:29.351624 24365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:29.470340 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24242
I20260928 05:26:29.487376 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24376
Sep 28 05:26:29 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:26:29.501730 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.488050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.8.124.194:38247)
0928 05:26:29.488218 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.488223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.488241 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:26:29.488437 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.488441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.488504 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.488591 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:29.489991 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.490716 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.491967 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.492362 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.492465 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.496564 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:29.496577 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:29.496593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:29.496625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:29.499199 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.499824 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.499830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.499833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.499943 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.500285 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.500288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.500291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.500652 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:29.500930 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.501192 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.501389 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:26:29.502411 24499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:29.588267 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24376
I20260928 05:26:29.601382 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24509
Sep 28 05:26:29 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:26:29.616076 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:29.602097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.8.124.195:36913)
0928 05:26:29.602301 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:29.602307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:29.602327 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:26:29.602463 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:29.602469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:29.602535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:29.602616 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:29.603876 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.604573 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.605959 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:29.606342 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:29.606448 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:29.610820 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:29.610834 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:29.610850 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:29.610890 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:29.613345 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.614100 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.614109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.614113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.614254 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:29.614604 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:29.614609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:29.614612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:29.614949 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:29.615211 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:29.615555 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 05:26:29.615772 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:26:29.616695 24632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:29.703565 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24509
I20260928 05:26:29.716988 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24149
I20260928 05:26:29.718111 24211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:29.825161 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24149
I20260928 05:26:49.842938 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--webserver_interface=127.8.124.254
--webserver_port=39215
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:26:50.157092 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:50.157368 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:50.157495 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:50.164685 24686 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:26:50.164773 24686 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:26:50.164812 24686 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:26:50.164880 24686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:26:50.164947 24686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:50.165040 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:50.165102 24686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:26:50.165174 24686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:26:50.174990 24686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:43739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=39215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:50.176275 24686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:50.177579 24686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:50.198678 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:50.198701 24693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:50.198678 24692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:50.199306 24686 server_base.cc:1034] running on GCE node
I20260928 05:26:50.200203 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:50.202065 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:50.203359 24686 hybrid_clock.cc:648] HybridClock initialized: now 1790573210203284 us; error 54 us; skew 500 ppm
I20260928 05:26:50.203765 24686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:50 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573210, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:50.211947 24686 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:26:50.213629 24686 webserver.cc:460] Webserver started at http://127.8.124.254:39215/ using document root <none> and password file <none>
I20260928 05:26:50.214457 24686 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:50.214576 24686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:50.220543 24686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:26:50.223603 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:50.225013 24686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:26:50.225195 24686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a9b416cf1f9c4223a51dc064d2675dc9"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "783ebdb31a9a431cd25f55cc4ba15cbd"
server_key_iv: "5c6091c3edccd405650e480cf0bfed03"
server_key_version: "encryptionkey@0"
I20260928 05:26:50.226334 24686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:50.240854 24686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:50.244829 24686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:50.245093 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:50.262212 24686 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:43739
I20260928 05:26:50.262252 24753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:43739 every 8 connection(s)
I20260928 05:26:50.263705 24686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:26:50.270548 24754 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:26:50.273759 24754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap starting.
I20260928 05:26:50.274178 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24686
I20260928 05:26:50.275139 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:35613
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=40077
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:26:50.295010 24754 log.cc:824] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:50.306726 24754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:26:50.307206 24754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap complete.
I20260928 05:26:50.315960 24754 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:50.316744 24754 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Initialized, Role: FOLLOWER
I20260928 05:26:50.317358 24754 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:50.317560 24754 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:26:50.317704 24754 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:26:50.317852 24754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:26:50.321393 24754 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:50.321887 24754 leader_election.cc:302] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9b416cf1f9c4223a51dc064d2675dc9; no voters:
I20260928 05:26:50.322458 24754 leader_election.cc:288] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:26:50.322597 24758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:26:50.323756 24758 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 LEADER]: Becoming Leader. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Running, Role: LEADER
I20260928 05:26:50.324401 24758 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:26:50.324783 24754 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:26:50.326916 24760 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9b416cf1f9c4223a51dc064d2675dc9. Latest consensus state: current_term: 2 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:26:50.327349 24760 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:50.327672 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:26:50.327991 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:26:50.334177 24765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:26:50.342343 24765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:26:50.343940 24765 catalog_manager.cc:1259] Loaded cluster ID: 81c4b66ee3734532a8a2680bb1b69ede
I20260928 05:26:50.344091 24765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:26:50.348269 24765 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:26:50.349961 24765 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Loaded TSK: 0
I20260928 05:26:50.383859 24765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Generated new TSK 1
I20260928 05:26:50.384241 24765 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:26:50.628674 24756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:50.628916 24756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:50.629026 24756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:50.636339 24756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:50.636442 24756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:50.636575 24756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:26:50.646107 24756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:35613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=40077
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:50.647305 24756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:50.648547 24756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:50.662490 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:50.662744 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:50.662911 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:50.663345 24756 server_base.cc:1034] running on GCE node
I20260928 05:26:50.663902 24756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:50.664788 24756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:50.665978 24756 hybrid_clock.cc:648] HybridClock initialized: now 1790573210665916 us; error 55 us; skew 500 ppm
I20260928 05:26:50.666280 24756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:50 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573210, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:50.669209 24756 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:26:50.670646 24756 webserver.cc:460] Webserver started at http://127.8.124.193:40077/ using document root <none> and password file <none>
I20260928 05:26:50.671275 24756 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:50.671391 24756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:50.676507 24756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:26:50.679070 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:50.680312 24756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:26:50.680444 24756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "26a2c5ead44b4c17837e9e1452d2836b"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "fcf4aaf9620e05fff699cc1330fed9e8"
server_key_iv: "e2f11e22b07593af365e3811d9023d8f"
server_key_version: "encryptionkey@0"
I20260928 05:26:50.681329 24756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:50.700794 24756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:50.705008 24756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:50.705273 24756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:50.706373 24756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:50.707790 24756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:50.707888 24756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:50.707993 24756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:50.708065 24756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:50.739205 24756 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:35613
I20260928 05:26:50.739244 24903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:35613 every 8 connection(s)
I20260928 05:26:50.740543 24756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:26:50.750149 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24756
I20260928 05:26:50.750742 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:38247
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=41515
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:26:50 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573210, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:50.763149 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:50.743482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:40401)
0928 05:26:50.744277 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:50.744322 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:50.746078 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:50.747378 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:50.747385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:50.747862 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:50.748422 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:50.748446 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:50.750031 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:50.750035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:50.751370 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:50.751382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:50.752401 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:50.752408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:50.752684 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:50.753346 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:50.753377 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:50.756071 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:50.759505 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:50.759511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:50.759522 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:50.759932 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:50.760367 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:50.760370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:50.760373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:50.760529 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:50.761311 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:50.761316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:50.761593 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 05:26:50.761865 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 05:26:50.762225 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":288,"threads_started":1}
I20260928 05:26:50.763583 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:50.743301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:40401 (local address 127.8.124.254:43739)
0928 05:26:50.743833 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:50.743848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:50.744444 (+ 596us) server_negotiation.cc:407] Connection header received
0928 05:26:50.746586 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:50.746622 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:50.746949 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:50.747237 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:50.748598 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:50.749873 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:50.751506 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:50.752150 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:50.752321 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:50.756352 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:50.756361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:50.756377 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:50.756417 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:50.759379 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:50.760048 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:50.760057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:50.760065 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:50.760267 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:50.760612 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:50.760617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:50.760620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:50.761199 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:50.761369 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:50.762066 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 05:26:50.762451 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":216,"threads_started":1}
I20260928 05:26:50.766232 24904 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:50.766527 24904 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:50.767318 24904 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
I20260928 05:26:50.769697 24719 ts_manager.cc:194] Registered new tserver with Master: 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613)
I20260928 05:26:50.771801 24719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:40401
W20260928 05:26:51.071671 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:51.071945 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:51.072038 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:51.079169 24908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:51.079273 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:51.079397 24908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:26:51.088953 24908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:38247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=41515
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:51.090191 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:51.091413 24908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:51.103967 24913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:51.104017 24916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:51.103998 24914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:51.104657 24908 server_base.cc:1034] running on GCE node
I20260928 05:26:51.105322 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:51.106235 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:51.107438 24908 hybrid_clock.cc:648] HybridClock initialized: now 1790573211107388 us; error 42 us; skew 500 ppm
I20260928 05:26:51.107741 24908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:51 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573211, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:51.110669 24908 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:26:51.112044 24908 webserver.cc:460] Webserver started at http://127.8.124.194:41515/ using document root <none> and password file <none>
I20260928 05:26:51.112675 24908 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:51.112788 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:51.117937 24908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:26:51.120497 24923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.121822 24908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:26:51.121968 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "af3e8642856b4aa79a25c7488b63f720"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "04d5066e61a98500d2db90deea8a0d4f"
server_key_iv: "699bf68b501c48b58ed9fd9d044617f4"
server_key_version: "encryptionkey@0"
I20260928 05:26:51.122881 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:51.148733 24908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:51.152930 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:51.153188 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:51.154208 24908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:51.155586 24908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:51.155680 24908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.155787 24908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:51.155853 24908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.189256 24908 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:38247
I20260928 05:26:51.189306 25035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:38247 every 8 connection(s)
I20260928 05:26:51.190699 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:26:51.196075 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 24908
I20260928 05:26:51.196883 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:36913
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=45201
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:26:51 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573211, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:51.210871 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.193067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:37591 (local address 127.8.124.254:43739)
0928 05:26:51.193250 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.193255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.194566 (+ 1311us) server_negotiation.cc:407] Connection header received
0928 05:26:51.196445 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.196451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.196542 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.196653 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:51.198227 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.199170 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.200773 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.201387 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.201504 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.205114 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:51.205119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:51.205122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:51.205157 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:51.207653 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.208517 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.208521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.208524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.208653 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.209173 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.209177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.209179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.209457 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:51.209571 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.210447 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.210636 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:26:51.212149 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.193594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:37591)
0928 05:26:51.194400 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.194451 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.196128 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.196887 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.196896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.197443 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:51.198058 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.198076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.199350 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.199354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.200628 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.200640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.201644 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.201652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.201881 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.202536 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:51.202558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:51.204857 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:51.207791 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:51.207800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:51.207816 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:51.208369 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:51.208759 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:51.208763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:51.208767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:51.208989 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:51.209580 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:51.209588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:51.210015 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.210324 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.210829 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":297,"threads_started":1}
I20260928 05:26:51.213639 25036 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:51.213940 25036 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:51.214671 25036 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
I20260928 05:26:51.216120 24719 ts_manager.cc:194] Registered new tserver with Master: af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247)
I20260928 05:26:51.216920 24719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:37591
W20260928 05:26:51.510223 25039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:26:51.510499 25039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:26:51.510613 25039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:26:51.517643 25039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:26:51.517765 25039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:26:51.517921 25039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:26:51.527613 25039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:36913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=45201
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:26:51.528868 25039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:26:51.530216 25039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:26:51.542450 25044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:51.542408 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:26:51.542483 25045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:26:51.543149 25039 server_base.cc:1034] running on GCE node
I20260928 05:26:51.543705 25039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:26:51.544584 25039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:26:51.545759 25039 hybrid_clock.cc:648] HybridClock initialized: now 1790573211545714 us; error 36 us; skew 500 ppm
I20260928 05:26:51.546044 25039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:26:51 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573211, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:26:51.549346 25039 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:26:51.550932 25039 webserver.cc:460] Webserver started at http://127.8.124.195:45201/ using document root <none> and password file <none>
I20260928 05:26:51.551586 25039 fs_manager.cc:359] Metadata directory not provided
I20260928 05:26:51.551712 25039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:26:51.556854 25039 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:26:51.559396 25054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.560637 25039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:26:51.560791 25039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "e001b01d363dd01098dea4dcd5dcf05b"
server_key_iv: "835db6ab9450458cab87457ca5a28f85"
server_key_version: "encryptionkey@0"
I20260928 05:26:51.561774 25039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:26:51.576136 25039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:26:51.579641 25039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:26:51.579872 25039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:26:51.580746 25039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:26:51.582008 25039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:26:51.582103 25039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.582207 25039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:26:51.582279 25039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:26:51.610692 25039 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:36913
I20260928 05:26:51.610755 25166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:36913 every 8 connection(s)
I20260928 05:26:51.612056 25039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:26:51.620421 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25039
Sep 28 05:26:51 dist-test-slave-515x krb5kdc[24136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573211, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:26:51.632134 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.614450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:56593 (local address 127.8.124.254:43739)
0928 05:26:51.614614 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.614619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.615658 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 05:26:51.617462 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.617468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.617544 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.617652 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:51.619076 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.619809 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.621425 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.621937 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.622045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.625478 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:51.625483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:51.625486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:51.625523 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:51.628302 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.629014 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.629023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.629026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.629144 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.629568 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.629573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.629577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.629983 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:51.630132 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.631711 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.631908 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:26:51.632215 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.614851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:56593)
0928 05:26:51.615501 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.615542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.617200 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.617892 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.617901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.618376 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:26:51.618917 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.618934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.619954 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.619958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.621295 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.621307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.622139 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.622147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.622518 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.623107 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:26:51.623127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:26:51.625287 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:26:51.628433 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:51.628442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:51.628478 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:51.628909 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:51.629256 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:26:51.629261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:26:51.629265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:26:51.629434 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:26:51.630110 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:26:51.630116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:26:51.630415 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.630666 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.631102 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260928 05:26:51.633780 25167 heartbeater.cc:347] Connected to a master server at 127.8.124.254:43739
I20260928 05:26:51.634233 25167 heartbeater.cc:464] Registering TS with master...
I20260928 05:26:51.635207 25167 heartbeater.cc:511] Master 127.8.124.254:43739 requested a full tablet report, sending...
I20260928 05:26:51.636858 24719 ts_manager.cc:194] Registered new tserver with Master: d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913)
I20260928 05:26:51.637921 24719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:56593
I20260928 05:26:51.639844 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.621273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.8.124.254:43739)
0928 05:26:51.626003 (+ 4730us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.626008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.626349 (+ 341us) server_negotiation.cc:407] Connection header received
0928 05:26:51.626514 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.626518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.626575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.626676 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:51.628191 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.628999 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.630626 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.631041 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.631143 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.634203 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:51.634210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:51.634214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:51.634250 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:51.637130 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.637939 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.637942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.637944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.638076 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.638463 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.638466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.638468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.638738 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:51.638974 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.639293 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.639501 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4623,"thread_start_us":4486,"threads_started":1}
I20260928 05:26:51.642118 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:26:51.648092 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.642989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.8.124.254:43739)
0928 05:26:51.643143 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.643149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.643474 (+ 325us) server_negotiation.cc:407] Connection header received
0928 05:26:51.643664 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.643669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.643793 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.643906 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:51.644311 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.644955 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.646361 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.646763 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.646864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.647342 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:51.647512 (+ 170us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 05:26:51.647836 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52884: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 05:26:51.648188 25171 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52884: token expired
W20260928 05:26:51.648729 24640 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.124.254:43739)
I20260928 05:26:51.659685 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.649168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.8.124.254:43739)
0928 05:26:51.649304 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.649308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.649322 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:26:51.649362 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.649365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.649411 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.649481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:51.650768 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.651449 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.652665 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.653020 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.653103 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.655395 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:51.655399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:51.655402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:51.655435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:51.657682 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.658228 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.658232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.658234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.658340 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.658605 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.658608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.658610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.658874 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:51.659037 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.659272 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.659416 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:26:51.662763 24719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:26:51.669754 24719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:26:51.703529 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.680350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38247 (local address 127.0.0.1:43016)
0928 05:26:51.693793 (+ 13443us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.693843 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.694116 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.695309 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.695322 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.695356 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.695798 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.695810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.698304 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.698310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.700600 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.700611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.702606 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.702612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.702817 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.702870 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.703004 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.703169 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13158,"thread_start_us":14899,"threads_started":1}
I20260928 05:26:51.703572 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.680630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.8.124.194:38247)
0928 05:26:51.681276 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.681286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.693951 (+ 12665us) server_negotiation.cc:407] Connection header received
0928 05:26:51.694370 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.694380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.694658 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.694820 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.696013 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.698107 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.700754 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.702086 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.702207 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.702453 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.703068 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.703242 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":692,"threads_started":1}
I20260928 05:26:51.705507 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.680497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36913 (local address 127.0.0.1:52998)
0928 05:26:51.695023 (+ 14526us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.695043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.695484 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.696964 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.696969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.697074 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.697460 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.697471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.700121 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.700130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.702506 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.702641 (+ 135us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.704702 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.704709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.704923 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.704955 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.705028 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.705152 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14400,"mutex_wait_us":43,"thread_start_us":20401,"threads_started":1}
I20260928 05:26:51.705953 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.681086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.8.124.195:36913)
0928 05:26:51.693630 (+ 12544us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.693640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.695191 (+ 1551us) server_negotiation.cc:407] Connection header received
0928 05:26:51.695722 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.695731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.695974 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.697622 (+ 1648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.697922 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.699952 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.702861 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.704490 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.704612 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.705389 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.705469 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.705650 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12342,"thread_start_us":12564,"threads_started":1}
I20260928 05:26:51.707726 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.681069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:35613 (local address 127.0.0.1:46426)
0928 05:26:51.700656 (+ 19587us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.700674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.700798 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.701392 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.701397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.701422 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.701793 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.701805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.703585 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.703590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.705240 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.705253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.706983 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.706989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.707143 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.707189 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.707276 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.707386 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19450,"mutex_wait_us":3,"thread_start_us":12336,"threads_started":1}
I20260928 05:26:51.707814 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.682195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.8.124.193:35613)
0928 05:26:51.700189 (+ 17994us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.700198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.700863 (+ 665us) server_negotiation.cc:407] Connection header received
0928 05:26:51.700959 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.700966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.701209 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.701388 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.701952 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.703436 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.705392 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.706763 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.706890 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.707065 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.707345 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.707495 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17830,"thread_start_us":17723,"threads_started":1}
I20260928 05:26:51.707614 24971 tablet_service.cc:1467] Processing CreateTablet for tablet 53fe7c6f28114a468aa7abddd585743a (DEFAULT_TABLE table=test-table [id=483eecf1c3874b5b9cd344f6338cb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:51.709378 24971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fe7c6f28114a468aa7abddd585743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:51.709003 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 53fe7c6f28114a468aa7abddd585743a (DEFAULT_TABLE table=test-table [id=483eecf1c3874b5b9cd344f6338cb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:51.710453 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fe7c6f28114a468aa7abddd585743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:51.711150 24839 tablet_service.cc:1467] Processing CreateTablet for tablet 53fe7c6f28114a468aa7abddd585743a (DEFAULT_TABLE table=test-table [id=483eecf1c3874b5b9cd344f6338cb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:26:51.712379 24839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fe7c6f28114a468aa7abddd585743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:26:51.721035 25183 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Bootstrap starting.
I20260928 05:26:51.723780 25183 tablet_bootstrap.cc:654] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:51.725070 25183 log.cc:824] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:51.726174 25184 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Bootstrap starting.
I20260928 05:26:51.729923 25184 tablet_bootstrap.cc:654] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:51.731179 25184 log.cc:824] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:51.732674 25186 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Bootstrap starting.
I20260928 05:26:51.734228 25184 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: No bootstrap required, opened a new log
I20260928 05:26:51.734499 25184 ts_tablet_manager.cc:1397] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:26:51.735126 25183 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: No bootstrap required, opened a new log
I20260928 05:26:51.735358 25183 ts_tablet_manager.cc:1397] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260928 05:26:51.736179 25186 tablet_bootstrap.cc:654] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Neither blocks nor log segments found. Creating new log.
I20260928 05:26:51.737311 25186 log.cc:824] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:26:51.743357 25186 tablet_bootstrap.cc:492] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: No bootstrap required, opened a new log
I20260928 05:26:51.743161 25183 raft_consensus.cc:348] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.743650 25183 raft_consensus.cc:374] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:51.743757 25183 raft_consensus.cc:729] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3e8642856b4aa79a25c7488b63f720, State: Initialized, Role: FOLLOWER
I20260928 05:26:51.743937 25186 ts_tablet_manager.cc:1397] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 05:26:51.744467 25183 consensus_queue.cc:260] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.744781 25184 raft_consensus.cc:348] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.745242 25184 raft_consensus.cc:374] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:51.745334 25184 raft_consensus.cc:729] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f5e2b6a77e4aebad6055203c8cf310, State: Initialized, Role: FOLLOWER
I20260928 05:26:51.746001 25184 consensus_queue.cc:260] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.754261 25036 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:51.754712 25184 ts_tablet_manager.cc:1428] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260928 05:26:51.755020 25167 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:51.755049 25183 ts_tablet_manager.cc:1428] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
I20260928 05:26:51.756246 25186 raft_consensus.cc:348] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.756745 25186 raft_consensus.cc:374] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:26:51.756888 25186 raft_consensus.cc:729] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a2c5ead44b4c17837e9e1452d2836b, State: Initialized, Role: FOLLOWER
I20260928 05:26:51.757858 25186 consensus_queue.cc:260] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.758898 24904 heartbeater.cc:503] Master 127.8.124.254:43739 was elected leader, sending a full tablet report...
I20260928 05:26:51.759379 25186 ts_tablet_manager.cc:1428] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 05:26:51.826661 25189 raft_consensus.cc:482] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:26:51.826952 25189 raft_consensus.cc:504] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.828580 25189 leader_election.cc:288] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913), 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613)
I20260928 05:26:51.831427 25190 raft_consensus.cc:482] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:26:51.831760 25190 raft_consensus.cc:504] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.833328 25190 leader_election.cc:288] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247), 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613)
I20260928 05:26:51.836373 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.829060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:41447 (local address 127.8.124.193:35613)
0928 05:26:51.829292 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.829298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.829993 (+ 695us) server_negotiation.cc:407] Connection header received
0928 05:26:51.830130 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.830135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.830244 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.830352 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.830874 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.831862 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.833781 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.835098 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.835241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.835313 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.835936 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.836079 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:26:51.837080 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.829066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:35613 (local address 127.8.124.194:41447)
0928 05:26:51.829909 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.829922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.830016 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.830373 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.830378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.830404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.830719 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.830729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.832016 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.832021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.833548 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.833564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.835553 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.835563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.835751 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.835784 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.836585 (+ 801us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.836725 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":548,"threads_started":1}
I20260928 05:26:51.838055 24859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "af3e8642856b4aa79a25c7488b63f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a2c5ead44b4c17837e9e1452d2836b" is_pre_election: true
I20260928 05:26:51.838584 24859 raft_consensus.cc:2454] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3e8642856b4aa79a25c7488b63f720 in term 0.
I20260928 05:26:51.839337 24927 leader_election.cc:302] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a2c5ead44b4c17837e9e1452d2836b, af3e8642856b4aa79a25c7488b63f720; no voters:
I20260928 05:26:51.839728 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.828906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:37819 (local address 127.8.124.195:36913)
0928 05:26:51.829096 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.829101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.829908 (+ 807us) server_negotiation.cc:407] Connection header received
0928 05:26:51.830181 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.830186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.830261 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.830400 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.831571 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.834592 (+ 3021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.837490 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.838746 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.838865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.838929 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.839369 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.839489 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:26:51.839785 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.828974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:36913 (local address 127.8.124.194:37819)
0928 05:26:51.829789 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.829808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.830001 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.830563 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.830567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.830588 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.831408 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.831422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.835358 (+ 3936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.835363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.837332 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.837346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.839102 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.839106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.839227 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.839249 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.839329 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.839449 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":338,"threads_started":1}
I20260928 05:26:51.839780 25189 raft_consensus.cc:2781] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:26:51.839952 25189 raft_consensus.cc:482] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:26:51.840049 25189 raft_consensus.cc:3037] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:51.840561 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "af3e8642856b4aa79a25c7488b63f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" is_pre_election: true
I20260928 05:26:51.840890 25122 raft_consensus.cc:2454] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3e8642856b4aa79a25c7488b63f720 in term 0.
I20260928 05:26:51.842898 25189 raft_consensus.cc:504] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.843793 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.834180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:37719 (local address 127.8.124.194:38247)
0928 05:26:51.834328 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.834334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.835476 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 05:26:51.835727 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.835732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.835805 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.835864 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.836545 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.837672 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.841565 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.842862 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.843017 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.843083 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.843458 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.843579 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:26:51.843856 25122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "af3e8642856b4aa79a25c7488b63f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
I20260928 05:26:51.844076 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.833648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:35613 (local address 127.8.124.195:33667)
0928 05:26:51.833939 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.833957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.834056 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.835517 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.835522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.835602 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.835804 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.835815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.837913 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.837917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.839664 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.839679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.843224 (+ 3545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.843229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.843463 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.843486 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.843586 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.843728 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":74,"spinlock_wait_cycles":21760}
I20260928 05:26:51.844050 25122 raft_consensus.cc:3037] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:51.844175 24859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "af3e8642856b4aa79a25c7488b63f720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a2c5ead44b4c17837e9e1452d2836b"
I20260928 05:26:51.844383 24859 raft_consensus.cc:3037] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:26:51.844647 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.833743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38247 (local address 127.8.124.195:37719)
0928 05:26:51.835393 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0928 05:26:51.835410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:26:51.835557 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:26:51.836030 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:26:51.836034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:26:51.836057 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:26:51.836385 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.836396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.837856 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.837861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.841414 (+ 3553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:26:51.841429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.843059 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.843065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:26:51.843266 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.843301 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:26:51.843979 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 05:26:51.844252 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":3,"thread_start_us":1320,"threads_started":1}
I20260928 05:26:51.844726 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.834361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:33667 (local address 127.8.124.193:35613)
0928 05:26:51.835218 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.835223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.835246 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:26:51.835303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.835307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.835371 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.835924 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:26:51.836214 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.837776 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.841823 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.843023 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.843335 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.843403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.844236 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.844363 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":479,"threads_started":1}
I20260928 05:26:51.845398 24858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a2c5ead44b4c17837e9e1452d2836b" is_pre_election: true
I20260928 05:26:51.845690 24991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fe7c6f28114a468aa7abddd585743a" candidate_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3e8642856b4aa79a25c7488b63f720" is_pre_election: true
I20260928 05:26:51.846130 24991 raft_consensus.cc:2379] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0f5e2b6a77e4aebad6055203c8cf310 in current term 1: Already voted for candidate af3e8642856b4aa79a25c7488b63f720 in this term.
I20260928 05:26:51.847144 25122 raft_consensus.cc:2454] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3e8642856b4aa79a25c7488b63f720 in term 1.
I20260928 05:26:51.847106 25056 leader_election.cc:302] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f5e2b6a77e4aebad6055203c8cf310; no voters: 26a2c5ead44b4c17837e9e1452d2836b, af3e8642856b4aa79a25c7488b63f720
I20260928 05:26:51.847527 25190 raft_consensus.cc:2726] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:26:51.847636 24926 leader_election.cc:302] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af3e8642856b4aa79a25c7488b63f720, d0f5e2b6a77e4aebad6055203c8cf310; no voters:
I20260928 05:26:51.847829 24859 raft_consensus.cc:2454] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3e8642856b4aa79a25c7488b63f720 in term 1.
I20260928 05:26:51.848085 25189 leader_election.cc:288] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [CANDIDATE]: Term 1 election: Requested vote from peers d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913), 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613)
I20260928 05:26:51.848284 25189 raft_consensus.cc:2781] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:26:51.848417 25189 raft_consensus.cc:686] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 LEADER]: Becoming Leader. State: Replica: af3e8642856b4aa79a25c7488b63f720, State: Running, Role: LEADER
I20260928 05:26:51.848963 25189 consensus_queue.cc:237] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } }
I20260928 05:26:51.853696 24717 catalog_manager.cc:5495] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3e8642856b4aa79a25c7488b63f720 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "af3e8642856b4aa79a25c7488b63f720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3e8642856b4aa79a25c7488b63f720" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 } health_report { overall_health: UNKNOWN } } }
W20260928 05:26:51.863524 25168 tablet.cc:2369] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:26:51.880316 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.875338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.8.124.194:38247)
0928 05:26:51.875492 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.875497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.875565 (+ 68us) server_negotiation.cc:407] Connection header received
0928 05:26:51.875738 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.875742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.875825 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.875947 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:26:51.876364 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.877100 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.878430 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.878850 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.878937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.879373 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:26:51.879868 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:26:51.879973 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.880018 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.880082 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:26:51.888044 24859 raft_consensus.cc:1261] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Refusing update from remote peer af3e8642856b4aa79a25c7488b63f720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:51.888096 25122 raft_consensus.cc:1261] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Refusing update from remote peer af3e8642856b4aa79a25c7488b63f720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:26:51.888936 25196 consensus_queue.cc:1035] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f5e2b6a77e4aebad6055203c8cf310" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:51.889274 25189 consensus_queue.cc:1035] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a2c5ead44b4c17837e9e1452d2836b" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:26:51.900243 25199 mvcc.cc:205] Tried to move back new op lower bound from 7334187875884388352 to 7334187875738165248. Current Snapshot: MvccSnapshot[applied={T|T < 7334187875884388352}]
I20260928 05:26:51.913365 24717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:51.913640 24717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:26:51.916790 24717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Sending DeleteTablet for 3 replicas of tablet 53fe7c6f28114a468aa7abddd585743a
I20260928 05:26:51.917887 25102 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fe7c6f28114a468aa7abddd585743a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:51 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:52998
I20260928 05:26:51.917977 24971 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fe7c6f28114a468aa7abddd585743a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:51 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:43016
I20260928 05:26:51.918192 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24756
I20260928 05:26:51.918828 24839 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fe7c6f28114a468aa7abddd585743a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:26:51 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:46426
I20260928 05:26:51.919109 25209 tablet_replica.cc:330] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: stopping tablet replica
I20260928 05:26:51.919499 25209 raft_consensus.cc:2229] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:26:51.920208 25209 raft_consensus.cc:2258] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:51.920935 25210 tablet_replica.cc:330] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: stopping tablet replica
I20260928 05:26:51.921195 25211 tablet_replica.cc:330] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: stopping tablet replica
I20260928 05:26:51.921371 25210 raft_consensus.cc:2229] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:51.921644 25211 raft_consensus.cc:2229] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:26:51.921900 25210 raft_consensus.cc:2258] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:51.922158 25211 raft_consensus.cc:2258] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:26:51.923439 25209 ts_tablet_manager.cc:1905] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:51.924001 25210 ts_tablet_manager.cc:1905] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:51.924168 25211 ts_tablet_manager.cc:1905] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:26:51.931051 25211 ts_tablet_manager.cc:1918] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:51.931226 25211 log.cc:1196] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/53fe7c6f28114a468aa7abddd585743a
I20260928 05:26:51.931234 25210 ts_tablet_manager.cc:1918] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:51.931528 25210 log.cc:1196] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/53fe7c6f28114a468aa7abddd585743a
I20260928 05:26:51.931716 25211 ts_tablet_manager.cc:1939] T 53fe7c6f28114a468aa7abddd585743a P 26a2c5ead44b4c17837e9e1452d2836b: Deleting consensus metadata
I20260928 05:26:51.932168 25210 ts_tablet_manager.cc:1939] T 53fe7c6f28114a468aa7abddd585743a P d0f5e2b6a77e4aebad6055203c8cf310: Deleting consensus metadata
I20260928 05:26:51.932901 25209 ts_tablet_manager.cc:1918] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:26:51.933055 25209 log.cc:1196] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/53fe7c6f28114a468aa7abddd585743a
I20260928 05:26:51.933579 24706 catalog_manager.cc:4841] TS 26a2c5ead44b4c17837e9e1452d2836b (127.8.124.193:35613): tablet 53fe7c6f28114a468aa7abddd585743a (table test-table [id=483eecf1c3874b5b9cd344f6338cb387]) successfully deleted
I20260928 05:26:51.933755 25209 ts_tablet_manager.cc:1939] T 53fe7c6f28114a468aa7abddd585743a P af3e8642856b4aa79a25c7488b63f720: Deleting consensus metadata
I20260928 05:26:51.934049 24705 catalog_manager.cc:4841] TS d0f5e2b6a77e4aebad6055203c8cf310 (127.8.124.195:36913): tablet 53fe7c6f28114a468aa7abddd585743a (table test-table [id=483eecf1c3874b5b9cd344f6338cb387]) successfully deleted
I20260928 05:26:51.935303 24704 catalog_manager.cc:4841] TS af3e8642856b4aa79a25c7488b63f720 (127.8.124.194:38247): tablet 53fe7c6f28114a468aa7abddd585743a (table test-table [id=483eecf1c3874b5b9cd344f6338cb387]) successfully deleted
I20260928 05:26:51.936712 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:51.919701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.8.124.193:35613)
0928 05:26:51.919886 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:51.919892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:51.919910 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:26:51.919960 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:51.919964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:51.920045 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:51.920135 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:51.922447 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.923538 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.925329 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:51.925971 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:51.926076 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:51.929591 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:51.929623 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:51.929639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:51.929677 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:51.933668 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.934591 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.934601 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.934604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.934747 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:51.935158 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:51.935163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:51.935167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:51.935590 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:51.935864 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:51.936179 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 05:26:51.936397 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:26:51.937328 24899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:52.025189 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24756
I20260928 05:26:52.040336 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24908
I20260928 05:26:52.054909 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:52.041076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.8.124.194:38247)
0928 05:26:52.041275 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:52.041281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:52.041302 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:26:52.041420 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:52.041426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:52.041511 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:52.041668 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:52.042972 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:52.044123 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:52.045323 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:52.046007 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:52.046144 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:52.048471 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:52.048485 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:52.048503 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:52.048544 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:52.052238 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:52.052878 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:52.052887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:52.052891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:52.053043 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:52.053353 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:52.053358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:52.053361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:52.053838 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:52.054179 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:52.054322 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 05:26:52.054540 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:26:52.055609 25031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:52.164822 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24908
I20260928 05:26:52.179677 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25039
I20260928 05:26:52.192440 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:26:52.180403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.8.124.195:36913)
0928 05:26:52.180558 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:26:52.180563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:26:52.180583 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:26:52.180717 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:26:52.180721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:26:52.180791 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:26:52.180874 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:26:52.182231 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:52.183006 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:52.184255 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:26:52.184655 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:26:52.184758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:26:52.187311 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:26:52.187326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:26:52.187343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:26:52.187376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:26:52.190017 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:52.190673 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:52.190683 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:52.190685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:52.190823 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:26:52.191101 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:26:52.191103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:26:52.191105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:26:52.191460 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:26:52.191663 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:26:52.191926 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:26:52.192126 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:26:52.193061 25162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:52.282323 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25039
I20260928 05:26:52.295567 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 24686
I20260928 05:26:52.296689 24749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:26:52.408804 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 24686
I20260928 05:27:12.426350 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--webserver_interface=127.8.124.254
--webserver_port=39215
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:12.741304 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:12.741619 25216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:12.741796 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:12.748981 25216 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:27:12.749090 25216 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:27:12.749146 25216 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:27:12.749213 25216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:12.749284 25216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:12.749374 25216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:12.749434 25216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:12.749506 25216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:12.759490 25216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:43739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=39215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:12.760818 25216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:12.762133 25216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:12.773649 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:12.773557 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:12.773614 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:12.773864 25216 server_base.cc:1034] running on GCE node
I20260928 05:27:12.774976 25216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:12.776418 25216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:12.777679 25216 hybrid_clock.cc:648] HybridClock initialized: now 1790573232777576 us; error 49 us; skew 500 ppm
I20260928 05:27:12.778002 25216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:12 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573232, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:12.786526 25216 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:27:12.788044 25216 webserver.cc:460] Webserver started at http://127.8.124.254:39215/ using document root <none> and password file <none>
I20260928 05:27:12.788733 25216 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:12.788825 25216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:12.794205 25216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:27:12.796931 25231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:12.798326 25216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:12.798475 25216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a9b416cf1f9c4223a51dc064d2675dc9"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "783ebdb31a9a431cd25f55cc4ba15cbd"
server_key_iv: "5c6091c3edccd405650e480cf0bfed03"
server_key_version: "encryptionkey@0"
I20260928 05:27:12.799399 25216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:12.822023 25216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:12.825923 25216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:12.826179 25216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:12.842628 25216 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:43739
I20260928 05:27:12.842658 25282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:43739 every 8 connection(s)
I20260928 05:27:12.843988 25216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:12.847761 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25216
I20260928 05:27:12.848706 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:35613
--local_ip_for_outbound_sockets=127.8.124.193
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=40077
--webserver_interface=127.8.124.193
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:27:12.852027 25283 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:27:12.856341 25283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap starting.
I20260928 05:27:12.875819 25283 log.cc:824] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:12.891180 25283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:27:12.891641 25283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Bootstrap complete.
I20260928 05:27:12.900513 25283 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:27:12.901400 25283 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Initialized, Role: FOLLOWER
I20260928 05:27:12.902073 25283 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:27:12.902314 25283 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:12.902421 25283 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:12.902596 25283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:27:12.906422 25283 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:27:12.906872 25283 leader_election.cc:302] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9b416cf1f9c4223a51dc064d2675dc9; no voters:
I20260928 05:27:12.907487 25283 leader_election.cc:288] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:27:12.907625 25287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:27:12.908670 25287 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [term 3 LEADER]: Becoming Leader. State: Replica: a9b416cf1f9c4223a51dc064d2675dc9, State: Running, Role: LEADER
I20260928 05:27:12.909232 25287 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } }
I20260928 05:27:12.909497 25283 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:12.911629 25288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:27:12.911974 25288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:12.912256 25289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9b416cf1f9c4223a51dc064d2675dc9. Latest consensus state: current_term: 3 leader_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b416cf1f9c4223a51dc064d2675dc9" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 43739 } } }
I20260928 05:27:12.912580 25289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:12.922737 25293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:12.932122 25293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:12.933755 25293 catalog_manager.cc:1259] Loaded cluster ID: 81c4b66ee3734532a8a2680bb1b69ede
I20260928 05:27:12.933938 25293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:12.938190 25293 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:12.939857 25293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Loaded TSK: 0
I20260928 05:27:12.940073 25293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9b416cf1f9c4223a51dc064d2675dc9: Loaded TSK: 1
F20260928 05:27:12.941228 25293 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f244c1099ad google::LogMessage::Fail() at ??:0
@ 0x7f244c10c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f244c1094de google::LogMessage::Flush() at ??:0
@ 0x7f244c10aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f244fe6d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f244fe5c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2461b381c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2461b37783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2461bc32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f2461bc30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f2461c0d5e9 std::function<>::operator()() at ??:0
@ 0x7f2461b3bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2461b39c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2461bc1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2461bc1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f244ed55345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f244ed5891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f244ed58799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f244ed33eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f245200d6db start_thread at ??:0
@ 0x7f244afa361f clone at ??:0
W20260928 05:27:13.205397 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:13.205724 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:13.205873 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:13.213042 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:13.213171 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:13.213327 25285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:13.223055 25285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:35613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=40077
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:13.224368 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:13.225728 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:13.238365 25330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:13.238421 25331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:13.238384 25333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:13.239210 25285 server_base.cc:1034] running on GCE node
I20260928 05:27:13.239785 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:13.240684 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:13.241904 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790573233241838 us; error 47 us; skew 500 ppm
I20260928 05:27:13.242244 25285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:13 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:13.247709 25285 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:27:13.249114 25285 webserver.cc:460] Webserver started at http://127.8.124.193:40077/ using document root <none> and password file <none>
I20260928 05:27:13.249810 25285 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:13.249924 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:13.255064 25285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:27:13.257573 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.258843 25285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:27:13.258986 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "26a2c5ead44b4c17837e9e1452d2836b"
format_stamp: "Formatted at 2026-09-28 05:26:27 on dist-test-slave-515x"
server_key: "fcf4aaf9620e05fff699cc1330fed9e8"
server_key_iv: "e2f11e22b07593af365e3811d9023d8f"
server_key_version: "encryptionkey@0"
I20260928 05:27:13.259888 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:13.278084 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:13.282876 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:13.283176 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:13.284423 25285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:13.286028 25285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:13.286127 25285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.286257 25285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:13.286334 25285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.318804 25285 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:35613
I20260928 05:27:13.318815 25452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:35613 every 8 connection(s)
I20260928 05:27:13.320684 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:13.324079 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25285
I20260928 05:27:13.324807 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:38247
--local_ip_for_outbound_sockets=127.8.124.194
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=41515
--webserver_interface=127.8.124.194
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:27:13.325546 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.323519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:53189)
0928 05:27:13.324350 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.324559 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":309,"threads_started":1}
W20260928 05:27:13.326211 25453 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:43739 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:43739: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
I20260928 05:27:13.326992 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.326700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:56321)
0928 05:27:13.326828 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.326892 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:27:13.327783 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.327481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:36003)
0928 05:27:13.327607 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.327678 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:27:13.327989 25453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:27:13.636408 25456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:13.636667 25456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:13.636765 25456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:13.643680 25456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:13.643774 25456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:13.643890 25456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:13.653261 25456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:38247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=41515
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:13.654421 25456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:13.655620 25456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:13.667644 25462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:13.667728 25464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:13.667826 25461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:13.668324 25456 server_base.cc:1034] running on GCE node
I20260928 05:27:13.668848 25456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:13.669716 25456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:13.670915 25456 hybrid_clock.cc:648] HybridClock initialized: now 1790573233670859 us; error 46 us; skew 500 ppm
I20260928 05:27:13.671244 25456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:13 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:13.674075 25456 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:27:13.675313 25456 webserver.cc:460] Webserver started at http://127.8.124.194:41515/ using document root <none> and password file <none>
I20260928 05:27:13.675894 25456 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:13.675987 25456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:13.680932 25456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:27:13.683533 25471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.684864 25456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:27:13.684990 25456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "af3e8642856b4aa79a25c7488b63f720"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "04d5066e61a98500d2db90deea8a0d4f"
server_key_iv: "699bf68b501c48b58ed9fd9d044617f4"
server_key_version: "encryptionkey@0"
I20260928 05:27:13.685930 25456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:13.703788 25456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:13.707365 25456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:13.707566 25456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:13.708395 25456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:13.709538 25456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:13.709645 25456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.709740 25456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:13.709795 25456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:13.738399 25456 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:38247
I20260928 05:27:13.738441 25583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:38247 every 8 connection(s)
I20260928 05:27:13.739693 25456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:13.744086 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.742417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:36519)
0928 05:27:13.743067 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.743275 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":246,"threads_started":1}
W20260928 05:27:13.744700 25584 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:43739 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:43739: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
I20260928 05:27:13.745324 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25456
I20260928 05:27:13.745462 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.745120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:36081)
0928 05:27:13.745262 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.745338 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:27:13.746021 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:36913
--local_ip_for_outbound_sockets=127.8.124.195
--tserver_master_addrs=127.8.124.254:43739
--webserver_port=45201
--webserver_interface=127.8.124.195
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:27:13.746287 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:13.745953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:60243)
0928 05:27:13.746087 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:13.746169 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:27:13.746498 25584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:27:14.047655 25587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:14.047892 25587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:14.048023 25587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:14.055035 25587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:14.055135 25587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:14.055276 25587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:14.064661 25587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:36913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=45201
--tserver_master_addrs=127.8.124.254:43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:14.065968 25587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:14.067227 25587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:14.079207 25592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:14.079250 25595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:14.079180 25593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:14.080010 25587 server_base.cc:1034] running on GCE node
I20260928 05:27:14.080547 25587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:14.081406 25587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:14.082609 25587 hybrid_clock.cc:648] HybridClock initialized: now 1790573234082537 us; error 62 us; skew 500 ppm
I20260928 05:27:14.082888 25587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:14 dist-test-slave-515x krb5kdc[24136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573234, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:14.085664 25587 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:27:14.086946 25587 webserver.cc:460] Webserver started at http://127.8.124.195:45201/ using document root <none> and password file <none>
I20260928 05:27:14.087567 25587 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:14.087675 25587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:14.092669 25587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:27:14.095183 25602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:14.096387 25587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:27:14.096529 25587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "d0f5e2b6a77e4aebad6055203c8cf310"
format_stamp: "Formatted at 2026-09-28 05:26:28 on dist-test-slave-515x"
server_key: "e001b01d363dd01098dea4dcd5dcf05b"
server_key_iv: "835db6ab9450458cab87457ca5a28f85"
server_key_version: "encryptionkey@0"
I20260928 05:27:14.097466 25587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:14.125909 25587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:14.130256 25587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:14.130512 25587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:14.131487 25587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:14.132781 25587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:14.132879 25587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:14.132982 25587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:14.133049 25587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:14.162577 25587 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:36913
I20260928 05:27:14.162667 25714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:36913 every 8 connection(s)
I20260928 05:27:14.163993 25587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:14.166018 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25587
I20260928 05:27:14.168926 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:14.167058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:42351)
0928 05:27:14.167833 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:14.168059 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
W20260928 05:27:14.169523 25715 heartbeater.cc:644] Failed to heartbeat to 127.8.124.254:43739 (0 consecutive failures): Network error: Failed to ping master at 127.8.124.254:43739: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
I20260928 05:27:14.170344 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:14.170015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:33259)
0928 05:27:14.170178 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:14.170241 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:27:14.171104 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:14.170803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:58547)
0928 05:27:14.170932 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:14.171000 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:27:14.171325 25715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:27:14.329314 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:14.328571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:38837)
0928 05:27:14.329004 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:14.329102 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260928 05:27:14.747841 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:14.747037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:59879)
0928 05:27:14.747516 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:14.747631 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260928 05:27:15.172643 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:15.171820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:60991)
0928 05:27:15.172314 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:15.172402 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":215,"threads_started":1}
I20260928 05:27:15.331210 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:15.330359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:54869)
0928 05:27:15.330890 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:15.330999 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260928 05:27:15.749270 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:15.748619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:59843)
0928 05:27:15.749027 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:15.749097 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":199,"threads_started":1}
I20260928 05:27:16.174283 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:16.173501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:55305)
0928 05:27:16.173980 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:16.174064 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260928 05:27:16.332907 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:16.332161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:52769)
0928 05:27:16.332614 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:16.332711 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260928 05:27:16.750860 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:16.750175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:53819)
0928 05:27:16.750590 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:16.750690 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":209,"threads_started":1}
I20260928 05:27:17.175781 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:17.175083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:60731)
0928 05:27:17.175531 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:17.175618 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260928 05:27:17.334547 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:17.333850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:58425)
0928 05:27:17.334266 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:17.334347 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":203,"threads_started":1}
I20260928 05:27:17.752564 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:17.751663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:35725)
0928 05:27:17.752287 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:17.752375 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":258,"threads_started":1}
I20260928 05:27:18.177371 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:18.176622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:59367)
0928 05:27:18.177044 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:18.177127 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260928 05:27:18.336086 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:18.335419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:38731)
0928 05:27:18.335846 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:18.335922 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":209,"threads_started":1}
I20260928 05:27:18.754127 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:18.753429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:40377)
0928 05:27:18.753874 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:18.753964 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260928 05:27:19.179052 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:19.178337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:53957)
0928 05:27:19.178788 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:19.178870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260928 05:27:19.337634 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:19.336933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:57685)
0928 05:27:19.337351 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:19.337437 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260928 05:27:19.755719 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:19.754965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:41109)
0928 05:27:19.755448 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:19.755540 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260928 05:27:20.180716 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:20.179968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:58751)
0928 05:27:20.180447 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:20.180550 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260928 05:27:20.339269 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:20.338501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:37521)
0928 05:27:20.338973 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:20.339051 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":215,"threads_started":1}
I20260928 05:27:20.757556 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:20.756723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:38889)
0928 05:27:20.757227 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:20.757347 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":225,"threads_started":1}
I20260928 05:27:21.182583 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:21.181584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:49099)
0928 05:27:21.182180 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:21.182297 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":235,"threads_started":1}
I20260928 05:27:21.340986 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:21.340211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:41365)
0928 05:27:21.340712 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:21.340801 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260928 05:27:21.759188 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:21.758418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:49595)
0928 05:27:21.758850 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:21.758943 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260928 05:27:22.184391 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:22.183547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:60465)
0928 05:27:22.184077 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:22.184172 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":299,"threads_started":1}
I20260928 05:27:22.342777 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:22.341953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:37819)
0928 05:27:22.342475 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:22.342557 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260928 05:27:22.760778 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:22.760058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:34263)
0928 05:27:22.760520 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:22.760605 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260928 05:27:23.186112 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:23.185251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:57697)
0928 05:27:23.185789 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:23.185925 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260928 05:27:23.344558 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:23.343713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:41289)
0928 05:27:23.344224 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:23.344344 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260928 05:27:23.762404 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:23.761720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:55943)
0928 05:27:23.762149 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:23.762234 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260928 05:27:24.187738 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:24.187029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:43283)
0928 05:27:24.187475 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:24.187561 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260928 05:27:24.346427 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:24.345672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:34441)
0928 05:27:24.346126 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:24.346220 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":218,"threads_started":1}
I20260928 05:27:24.764002 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:24.763273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:59457)
0928 05:27:24.763697 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:24.763778 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":205,"threads_started":1}
I20260928 05:27:25.189446 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:25.188658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:37739)
0928 05:27:25.189132 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:25.189229 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260928 05:27:25.347939 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:25.347277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:36595)
0928 05:27:25.347690 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:25.347771 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":209,"threads_started":1}
I20260928 05:27:25.765439 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:25.764788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:47587)
0928 05:27:25.765196 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:25.765281 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260928 05:27:26.190990 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:26.190343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:37809)
0928 05:27:26.190754 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:26.190836 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":197,"threads_started":1}
I20260928 05:27:26.349578 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:26.348811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:49345)
0928 05:27:26.349280 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:26.349374 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":215,"threads_started":1}
I20260928 05:27:26.767117 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:26.766371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:47651)
0928 05:27:26.766842 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:26.766949 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260928 05:27:27.192631 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:27.191839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:44973)
0928 05:27:27.192340 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:27.192436 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260928 05:27:27.351321 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:27.350585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:60057)
0928 05:27:27.351048 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:27.351123 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260928 05:27:27.768584 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:27.767927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:54527)
0928 05:27:27.768325 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:27.768400 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":207,"threads_started":1}
I20260928 05:27:28.194284 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:28.193520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:35847)
0928 05:27:28.194018 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:28.194105 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260928 05:27:28.353215 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:28.352406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.193:57613)
0928 05:27:28.352895 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:28.352993 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260928 05:27:28.770437 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:28.769697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.194:43349)
0928 05:27:28.770176 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:28.770266 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260928 05:27:29.175731 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25285
I20260928 05:27:29.196149 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:29.195369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:43739 (local address 127.8.124.195:57571)
0928 05:27:29.195819 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:29.195918 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.254:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260928 05:27:29.201025 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:29.176842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.8.124.193:35613)
0928 05:27:29.177485 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:29.177500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:29.177553 (+ 53us) server_negotiation.cc:407] Connection header received
0928 05:27:29.179270 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:29.179281 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:29.179850 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:29.180236 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:29.182090 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.183807 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.185681 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.186724 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.186922 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:29.194282 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:29.194291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:29.194309 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:29.194352 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:29.197760 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.198500 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.198507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.198515 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.198705 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.199120 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.199123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.199125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.199672 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:29.199913 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:29.200300 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 05:27:29.200736 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":299,"threads_started":1}
I20260928 05:27:29.202550 25448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:29.285698 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25285
I20260928 05:27:29.299216 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25456
I20260928 05:27:29.316360 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:29.300081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.8.124.194:38247)
0928 05:27:29.300650 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:29.300664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:29.300700 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:27:29.302489 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:29.302501 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:29.302972 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:29.303274 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:29.304621 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.305965 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.307261 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.307914 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.308037 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:29.310634 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:29.310641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:29.310657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:29.310693 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:29.313455 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.314111 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.314117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.314124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.314285 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.314635 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.314638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.314640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.315143 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:29.315370 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:29.315693 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 05:27:29.316084 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":292,"threads_started":1}
I20260928 05:27:29.317780 25579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:29.401175 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25456
I20260928 05:27:29.415333 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25587
I20260928 05:27:29.432909 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:29.416300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.8.124.195:36913)
0928 05:27:29.416912 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:29.416925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:29.416968 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:27:29.418644 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:29.418655 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:29.419110 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:29.419405 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:29.420736 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.422052 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.423287 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:29.423993 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:29.424132 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:29.426736 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:29.426747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:29.426764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:29.426804 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:29.429760 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.430536 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.430544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.430550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.430746 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:29.431063 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:29.431066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:29.431068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:29.431618 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:29.431845 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:29.432193 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 05:27:29.432603 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260928 05:27:29.434335 25710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:29.517181 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25587
2026-09-28T05:27:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63800 ms)
[----------] 30 tests from SecurityITest (283610 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:29 dist-test-slave-515x krb5kdc[25772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:29 dist-test-slave-515x krb5kdc[25772](info): set up 2 sockets
Sep 28 05:27:29 dist-test-slave-515x krb5kdc[25772](info): commencing operation
krb5kdc: starting...
W20260928 05:27:30.521479 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:30 dist-test-slave-515x krb5kdc[25772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:27:30.752823 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:35907
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:31.069504 25785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:31.069799 25785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:31.069886 25785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:31.077359 25785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:31.077468 25785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:31.077519 25785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:31.077574 25785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:31.077634 25785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:31.087353 25785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:31.088585 25785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:31.089953 25785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:31.102114 25791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:31.102072 25793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:31.102136 25790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:31.102531 25785 server_base.cc:1034] running on GCE node
I20260928 05:27:31.103550 25785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:31.104875 25785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:31.106072 25785 hybrid_clock.cc:648] HybridClock initialized: now 1790573251106014 us; error 39 us; skew 500 ppm
I20260928 05:27:31.106431 25785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:31 dist-test-slave-515x krb5kdc[25772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573251, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:31.109368 25785 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:27:31.110831 25785 webserver.cc:460] Webserver started at http://127.8.124.254:43309/ using document root <none> and password file <none>
I20260928 05:27:31.111510 25785 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:31.111615 25785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:31.111910 25785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:31.114913 25785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "83941ffe617c4b089b0d893c95d0439c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "3c9bf27e471feba5227bf7cfc6873147"
server_key_iv: "065e819da4a777a3caff164bec5d3ea7"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.115710 25785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "83941ffe617c4b089b0d893c95d0439c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "3c9bf27e471feba5227bf7cfc6873147"
server_key_iv: "065e819da4a777a3caff164bec5d3ea7"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.120810 25785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 05:27:31.124140 25800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:31.125438 25785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:27:31.125594 25785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "83941ffe617c4b089b0d893c95d0439c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "3c9bf27e471feba5227bf7cfc6873147"
server_key_iv: "065e819da4a777a3caff164bec5d3ea7"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.125834 25785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:31.148718 25785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:31.152621 25785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:31.152882 25785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:31.169047 25785 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:35907
I20260928 05:27:31.169070 25851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:35907 every 8 connection(s)
I20260928 05:27:31.170408 25785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:31.172633 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25785
I20260928 05:27:31.172809 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:31.173442 8691 external_mini_cluster.cc:1428] Setting key 16b1d8546d35c18f0851dde5ecad1b6d
I20260928 05:27:31.174297 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:27:31 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:31.189462 25852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: Bootstrap starting.
I20260928 05:27:31.193110 25852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:31.194360 25852 log.cc:824] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:31.197335 25852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: No bootstrap required, opened a new log
I20260928 05:27:31.197993 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:31.175279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.8.124.254:35907)
0928 05:27:31.176031 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:31.176047 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:31.176095 (+ 48us) server_negotiation.cc:407] Connection header received
0928 05:27:31.177943 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:31.177972 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:31.178491 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:31.178949 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:31.180589 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.182639 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.184468 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.185461 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.185658 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:31.190735 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:31.190743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:31.190758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:31.190804 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:31.193818 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:31.194693 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:31.194700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:31.194706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:31.194876 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:31.195258 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:31.195264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:31.195267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:31.195773 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:31.196026 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:31.196518 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 05:27:31.196938 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
I20260928 05:27:31.205832 25852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } }
I20260928 05:27:31.206231 25852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:31.206343 25852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83941ffe617c4b089b0d893c95d0439c, State: Initialized, Role: FOLLOWER
I20260928 05:27:31.206935 25852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } }
I20260928 05:27:31.207152 25852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:31.207283 25852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:31.207439 25852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:31.210028 25852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } }
I20260928 05:27:31.210554 25852 leader_election.cc:302] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83941ffe617c4b089b0d893c95d0439c; no voters:
I20260928 05:27:31.211099 25852 leader_election.cc:288] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:31.211221 25857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:31.211849 25857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [term 1 LEADER]: Becoming Leader. State: Replica: 83941ffe617c4b089b0d893c95d0439c, State: Running, Role: LEADER
I20260928 05:27:31.212349 25857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } }
I20260928 05:27:31.213068 25852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:31.214938 25858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83941ffe617c4b089b0d893c95d0439c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } } }
I20260928 05:27:31.215317 25858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:31.219105 25864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:31.218992 25859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83941ffe617c4b089b0d893c95d0439c. Latest consensus state: current_term: 1 leader_uuid: "83941ffe617c4b089b0d893c95d0439c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83941ffe617c4b089b0d893c95d0439c" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 35907 } } }
I20260928 05:27:31.219281 25859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:31.225261 25864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:31.234768 25864 catalog_manager.cc:1347] Generated new cluster ID: 78cfa59d28304c1aa13e6eb1bb6d6a65
I20260928 05:27:31.234895 25864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:31.253769 25864 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:31.255635 25864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:31.267817 25864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: Generated new TSK 0
I20260928 05:27:31.268743 25864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:27:31.340781 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:31.690414 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:31.690698 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:31.690824 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:31.698119 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:31.698233 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:31.698377 25878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:31.707818 25878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:31.709064 25878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:31.710404 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:31.722798 25883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:31.722833 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:31.722761 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:31.723919 25878 server_base.cc:1034] running on GCE node
I20260928 05:27:31.724514 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:31.725483 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:31.726673 25878 hybrid_clock.cc:648] HybridClock initialized: now 1790573251726618 us; error 50 us; skew 500 ppm
I20260928 05:27:31.726950 25878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:31 dist-test-slave-515x krb5kdc[25772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573251, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:31.729939 25878 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:27:31.731454 25878 webserver.cc:460] Webserver started at http://127.8.124.193:37493/ using document root <none> and password file <none>
I20260928 05:27:31.732056 25878 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:31.732161 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:31.732412 25878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:31.735289 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "7a8a294b40c449deb9520f5c9fe0b86c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "f8704c2b34307908c5e6d11c5f895fff"
server_key_iv: "f438303114f99435415d41782ee0363a"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.735997 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "7a8a294b40c449deb9520f5c9fe0b86c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "f8704c2b34307908c5e6d11c5f895fff"
server_key_iv: "f438303114f99435415d41782ee0363a"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.740787 25878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 05:27:31.744024 25893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:31.745355 25878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:27:31.745515 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "7a8a294b40c449deb9520f5c9fe0b86c"
format_stamp: "Formatted at 2026-09-28 05:27:31 on dist-test-slave-515x"
server_key: "f8704c2b34307908c5e6d11c5f895fff"
server_key_iv: "f438303114f99435415d41782ee0363a"
server_key_version: "encryptionkey@0"
I20260928 05:27:31.745738 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:31.761423 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:31.765223 25878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:31.765463 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:31.766413 25878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:31.767699 25878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:31.767769 25878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:31.767863 25878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:31.767899 25878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:31.796536 25878 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:37357
I20260928 05:27:31.796583 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:37357 every 8 connection(s)
I20260928 05:27:31.797809 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:31.806617 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 25878
I20260928 05:27:31.806802 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:31.807417 8691 external_mini_cluster.cc:1428] Setting key d25a66011e1a5322efccfb3675a375d5
Sep 28 05:27:31 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573251, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:31.816704 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:31.800268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:48661 (local address 127.8.124.254:35907)
0928 05:27:31.800687 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:31.800692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:31.801510 (+ 818us) server_negotiation.cc:407] Connection header received
0928 05:27:31.803322 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:31.803327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:31.803402 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:31.803494 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:31.804932 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.805695 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.807243 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.807667 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.807778 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:31.811103 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:31.811115 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:31.811118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:31.811151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:31.813788 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:31.814496 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:31.814502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:31.814505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:31.814625 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:31.815027 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:31.815032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:31.815036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:31.815306 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:31.815435 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:31.816134 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 05:27:31.816357 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260928 05:27:31.817713 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:31.800675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35907 (local address 127.8.124.193:48661)
0928 05:27:31.801349 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:31.801388 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:31.803059 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:31.803732 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:31.803740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:31.804230 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:31.804765 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:31.804782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.805845 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.805848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:31.807117 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:31.807129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:31.807895 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:31.807902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:31.808116 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:31.808713 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:31.808740 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:31.810902 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:31.813914 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:31.813924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:31.813941 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:31.814380 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:31.814725 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:31.814729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:31.814733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:31.814919 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:31.815426 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:31.815434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:31.815771 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 05:27:31.816016 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 05:27:31.816575 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":259,"threads_started":1}
I20260928 05:27:31.819162 26006 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35907
I20260928 05:27:31.819470 26006 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:31.820240 26006 heartbeater.cc:511] Master 127.8.124.254:35907 requested a full tablet report, sending...
I20260928 05:27:31.822304 25817 ts_manager.cc:194] Registered new tserver with Master: 7a8a294b40c449deb9520f5c9fe0b86c (127.8.124.193:37357)
I20260928 05:27:31.823750 25817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:48661
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:27:31.857158 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:32.165710 26012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:32.165997 26012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:32.166141 26012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:32.173475 26012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:32.173614 26012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:32.173758 26012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:32.183377 26012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:32.184568 26012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:32.185873 26012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:32.198470 26018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:32.198539 26017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:32.198566 26020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:32.199070 26012 server_base.cc:1034] running on GCE node
I20260928 05:27:32.199668 26012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:32.200557 26012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:32.201742 26012 hybrid_clock.cc:648] HybridClock initialized: now 1790573252201682 us; error 48 us; skew 500 ppm
I20260928 05:27:32.202036 26012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:32 dist-test-slave-515x krb5kdc[25772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573252, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:32.204953 26012 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:27:32.206378 26012 webserver.cc:460] Webserver started at http://127.8.124.194:34273/ using document root <none> and password file <none>
I20260928 05:27:32.207016 26012 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:32.207129 26012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:32.207427 26012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:32.210255 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "81df71e931b74824a9610f23a7bdc127"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "b40982c77f085d3afccb410fbd8478ae"
server_key_iv: "fb4c8b78bd44cad95b55e353d09f955b"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.210992 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "81df71e931b74824a9610f23a7bdc127"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "b40982c77f085d3afccb410fbd8478ae"
server_key_iv: "fb4c8b78bd44cad95b55e353d09f955b"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.215987 26012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 05:27:32.219270 26027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.220525 26012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:27:32.220666 26012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "81df71e931b74824a9610f23a7bdc127"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "b40982c77f085d3afccb410fbd8478ae"
server_key_iv: "fb4c8b78bd44cad95b55e353d09f955b"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.220837 26012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:32.235570 26012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:32.239423 26012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:32.239663 26012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:32.240530 26012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:32.241724 26012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:32.241798 26012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.241910 26012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:32.241964 26012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.270551 26012 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:38207
I20260928 05:27:32.270610 26139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:38207 every 8 connection(s)
I20260928 05:27:32.271766 26012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:32.277842 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26012
I20260928 05:27:32.277999 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:32.278398 8691 external_mini_cluster.cc:1428] Setting key 9e23a8ed55227710d6e16b2597ae5284
Sep 28 05:27:32 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573252, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:32.291572 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.274143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:54875 (local address 127.8.124.254:35907)
0928 05:27:32.274307 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.274312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.275428 (+ 1116us) server_negotiation.cc:407] Connection header received
0928 05:27:32.277204 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.277209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.277293 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.277382 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:32.278954 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.279909 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.282017 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.282438 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.282533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.286145 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:32.286152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:32.286155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:32.286187 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:32.288631 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.289378 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.289382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.289384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.289507 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.289929 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.289932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.289934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.290206 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:32.290337 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.291013 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.291239 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:27:32.292523 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.274615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35907 (local address 127.8.124.194:54875)
0928 05:27:32.275267 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.275302 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.276949 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.277659 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.277668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.278205 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:32.278783 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.278801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.280057 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.280063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.281884 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.281900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.282637 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.282644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.282914 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.283561 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:32.283586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:32.285927 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:32.288770 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:32.288780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:32.288798 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:32.289257 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:32.289631 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:32.289636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:32.289640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:32.289818 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:32.290325 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:32.290333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:32.290656 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 05:27:32.290895 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 05:27:32.291424 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":251,"threads_started":1}
I20260928 05:27:32.293920 26140 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35907
I20260928 05:27:32.294204 26140 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:32.294894 26140 heartbeater.cc:511] Master 127.8.124.254:35907 requested a full tablet report, sending...
I20260928 05:27:32.296241 25817 ts_manager.cc:194] Registered new tserver with Master: 81df71e931b74824a9610f23a7bdc127 (127.8.124.194:38207)
I20260928 05:27:32.297034 25817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:54875
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:27:32.326728 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:32.631804 26145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:32.632046 26145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:32.632162 26145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:32.639362 26145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:32.639453 26145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:32.639602 26145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:32.648890 26145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:32.650164 26145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:32.651396 26145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:32.663587 26151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:32.663618 26153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:32.663553 26150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:32.664433 26145 server_base.cc:1034] running on GCE node
I20260928 05:27:32.664995 26145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:32.665907 26145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:32.667091 26145 hybrid_clock.cc:648] HybridClock initialized: now 1790573252667049 us; error 34 us; skew 500 ppm
I20260928 05:27:32.667378 26145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:32 dist-test-slave-515x krb5kdc[25772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573252, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:32.670327 26145 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:27:32.671669 26145 webserver.cc:460] Webserver started at http://127.8.124.195:33517/ using document root <none> and password file <none>
I20260928 05:27:32.672324 26145 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:32.672437 26145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:32.672724 26145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:32.675572 26145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "761a0af93f1ec80f2e77777f09240ede"
server_key_iv: "4997a7a19790e0aa1c320fa492cb4d7d"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.676528 26145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "761a0af93f1ec80f2e77777f09240ede"
server_key_iv: "4997a7a19790e0aa1c320fa492cb4d7d"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.682112 26145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:27:32.685451 26160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.686771 26145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:32.686918 26145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8"
format_stamp: "Formatted at 2026-09-28 05:27:32 on dist-test-slave-515x"
server_key: "761a0af93f1ec80f2e77777f09240ede"
server_key_iv: "4997a7a19790e0aa1c320fa492cb4d7d"
server_key_version: "encryptionkey@0"
I20260928 05:27:32.687094 26145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:32.703671 26145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:32.707649 26145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:32.707899 26145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:32.708798 26145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:32.710167 26145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:32.710244 26145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.710374 26145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:32.710426 26145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:32.739557 26145 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:46111
I20260928 05:27:32.739599 26272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:46111 every 8 connection(s)
I20260928 05:27:32.740760 26145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:32.746469 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26145
I20260928 05:27:32.746657 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:32.747077 8691 external_mini_cluster.cc:1428] Setting key 5c3020d31534e225045d5d55230e24f4
Sep 28 05:27:32 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573252, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:32.759757 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.743157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:46635 (local address 127.8.124.254:35907)
0928 05:27:32.743328 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.743334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.744526 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 05:27:32.746589 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.746596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.746678 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.746779 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:32.748429 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.749214 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.750825 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.751220 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.751333 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.754541 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:32.754545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:32.754549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:32.754580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:32.756886 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.757572 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.757576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.757579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.757730 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.758127 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.758132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.758135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.758388 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:32.758517 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.759206 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.759419 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:32.760739 26275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.743667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:35907 (local address 127.8.124.195:46635)
0928 05:27:32.744371 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.744409 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.746269 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.747039 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.747046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.747571 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:32.748275 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.748302 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.749356 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.749361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.750678 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.750691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.751433 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.751440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.751655 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.752236 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:32.752261 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:32.754350 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:32.757021 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:32.757030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:32.757046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:32.757449 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:32.757830 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:32.757834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:32.757838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:32.758006 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:32.758505 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:32.758513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:32.758829 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 05:27:32.759060 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 05:27:32.759551 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":271,"threads_started":1}
I20260928 05:27:32.762092 26273 heartbeater.cc:347] Connected to a master server at 127.8.124.254:35907
I20260928 05:27:32.762398 26273 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:32.763020 26273 heartbeater.cc:511] Master 127.8.124.254:35907 requested a full tablet report, sending...
I20260928 05:27:32.764314 25817 ts_manager.cc:194] Registered new tserver with Master: 1d01f94ac9a94d2b92e7c4c8788d83d8 (127.8.124.195:46111)
I20260928 05:27:32.765053 25817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:46635
I20260928 05:27:32.774176 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:32.789847 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.778284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.8.124.254:35907)
0928 05:27:32.778478 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.778484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.778843 (+ 359us) server_negotiation.cc:407] Connection header received
0928 05:27:32.779004 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.779009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.779075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.779148 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:32.780541 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.781223 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.782599 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.782977 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.783064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.785181 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:32.785185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:32.785189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:32.785219 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:32.787665 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.788278 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.788282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.788284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.788407 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:32.788667 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:32.788670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:32.788671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:32.788951 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:32.789124 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.789376 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.789540 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:32.793694 25817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:32.800324 25817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:32.827003 26006 heartbeater.cc:503] Master 127.8.124.254:35907 was elected leader, sending a full tablet report...
I20260928 05:27:32.833920 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.812413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.8.124.194:38207)
0928 05:27:32.822994 (+ 10581us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.823005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.825000 (+ 1995us) server_negotiation.cc:407] Connection header received
0928 05:27:32.825252 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.825260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.825551 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.825787 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:32.826721 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828753 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.831527 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.832691 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.832822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.833005 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.833438 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.833592 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10398,"thread_start_us":10251,"threads_started":1}
I20260928 05:27:32.834056 26285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.811646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:37357 (local address 127.0.0.1:52478)
0928 05:27:32.812744 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.812784 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.812997 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.825884 (+ 12887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.825891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.825929 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:32.826350 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.826364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828895 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.828900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.831235 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.831249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.832898 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.832905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.833095 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.833163 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 05:27:32.833270 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 05:27:32.833464 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":39,"thread_start_us":739,"threads_started":1}
I20260928 05:27:32.834441 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.811988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38207 (local address 127.0.0.1:47724)
0928 05:27:32.824688 (+ 12700us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.824710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.824876 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.826022 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.826026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.826050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:32.826552 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.826565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828913 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.828918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.831404 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.831415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.833138 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.833143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.833263 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.833285 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:27:32.833367 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 05:27:32.833495 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12577,"mutex_wait_us":26,"thread_start_us":12756,"threads_started":1}
I20260928 05:27:32.834678 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.813534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.8.124.195:46111)
0928 05:27:32.824224 (+ 10690us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.824233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.825867 (+ 1634us) server_negotiation.cc:407] Connection header received
0928 05:27:32.826032 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.826037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.826207 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.826357 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:32.827064 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828554 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.831249 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.832687 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.832811 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.833040 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.833986 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.834360 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10253,"thread_start_us":10309,"threads_started":1}
I20260928 05:27:32.834817 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.811935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.8.124.193:37357)
0928 05:27:32.825321 (+ 13386us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.825329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.825355 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:27:32.825445 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.825451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.825655 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.826554 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:32.826788 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828737 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.831389 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.832694 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.834110 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.834307 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:32.834369 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 05:27:32.834517 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13209,"thread_start_us":13173,"threads_started":1}
I20260928 05:27:32.834919 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.813215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:46111 (local address 127.0.0.1:57880)
0928 05:27:32.825753 (+ 12538us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.825770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.825890 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.826373 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.826377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.826409 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:32.826861 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.826874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.828712 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.828718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.831106 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.831118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.833184 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.833190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.833359 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:32.833388 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:27:32.833459 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 05:27:32.833560 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12413,"mutex_wait_us":4,"thread_start_us":11222,"threads_started":1}
I20260928 05:27:32.838016 26075 tablet_service.cc:1467] Processing CreateTablet for tablet 7382fb5fe9624207a67adf76d01008e1 (DEFAULT_TABLE table=test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:32.838013 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 7382fb5fe9624207a67adf76d01008e1 (DEFAULT_TABLE table=test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:32.839120 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7382fb5fe9624207a67adf76d01008e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:32.839119 26075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7382fb5fe9624207a67adf76d01008e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:32.843902 26208 tablet_service.cc:1467] Processing CreateTablet for tablet 7382fb5fe9624207a67adf76d01008e1 (DEFAULT_TABLE table=test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:32.845297 26208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7382fb5fe9624207a67adf76d01008e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:32.850267 26291 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Bootstrap starting.
I20260928 05:27:32.852216 26292 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Bootstrap starting.
I20260928 05:27:32.853857 26291 tablet_bootstrap.cc:654] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:32.855086 26291 log.cc:824] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:32.855486 26292 tablet_bootstrap.cc:654] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:32.856523 26292 log.cc:824] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:32.858120 26291 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: No bootstrap required, opened a new log
I20260928 05:27:32.858379 26291 ts_tablet_manager.cc:1397] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260928 05:27:32.859368 26292 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: No bootstrap required, opened a new log
I20260928 05:27:32.859620 26292 ts_tablet_manager.cc:1397] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 05:27:32.864445 26295 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Bootstrap starting.
I20260928 05:27:32.867110 26295 tablet_bootstrap.cc:654] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:32.867986 26295 log.cc:824] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:32.868872 26291 raft_consensus.cc:348] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.869304 26291 raft_consensus.cc:374] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:32.869419 26291 raft_consensus.cc:729] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81df71e931b74824a9610f23a7bdc127, State: Initialized, Role: FOLLOWER
I20260928 05:27:32.869443 26292 raft_consensus.cc:348] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.869871 26292 raft_consensus.cc:374] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:32.869966 26292 raft_consensus.cc:729] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8a294b40c449deb9520f5c9fe0b86c, State: Initialized, Role: FOLLOWER
I20260928 05:27:32.870096 26291 consensus_queue.cc:260] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.870693 26295 tablet_bootstrap.cc:492] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: No bootstrap required, opened a new log
I20260928 05:27:32.870572 26292 consensus_queue.cc:260] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.870945 26295 ts_tablet_manager.cc:1397] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 05:27:32.871358 26140 heartbeater.cc:503] Master 127.8.124.254:35907 was elected leader, sending a full tablet report...
I20260928 05:27:32.871831 26291 ts_tablet_manager.cc:1428] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260928 05:27:32.873793 26292 ts_tablet_manager.cc:1428] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260928 05:27:32.879992 26295 raft_consensus.cc:348] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.880329 26295 raft_consensus.cc:374] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:32.880427 26295 raft_consensus.cc:729] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d01f94ac9a94d2b92e7c4c8788d83d8, State: Initialized, Role: FOLLOWER
I20260928 05:27:32.880949 26295 consensus_queue.cc:260] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.881877 26273 heartbeater.cc:503] Master 127.8.124.254:35907 was elected leader, sending a full tablet report...
I20260928 05:27:32.882277 26295 ts_tablet_manager.cc:1428] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260928 05:27:32.992285 26274 tablet.cc:2369] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:32.992442 26297 raft_consensus.cc:482] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:32.992730 26297 raft_consensus.cc:504] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:32.994431 26297 leader_election.cc:288] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8a294b40c449deb9520f5c9fe0b86c (127.8.124.193:37357), 1d01f94ac9a94d2b92e7c4c8788d83d8 (127.8.124.195:46111)
I20260928 05:27:33.001269 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.994835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:50139 (local address 127.8.124.193:37357)
0928 05:27:32.995007 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.995012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.995497 (+ 485us) server_negotiation.cc:407] Connection header received
0928 05:27:32.995723 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.995729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.995809 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.995924 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:32.996504 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.997444 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.999391 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.000347 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.000470 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.000523 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.000893 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.001026 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:27:33.001289 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.994737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:37357 (local address 127.8.124.194:50139)
0928 05:27:32.995406 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.995425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.995565 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.995958 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.995964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.995995 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:32.996366 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.996376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.997562 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.997567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.999228 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.999240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.000543 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.000549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:33.000710 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.000745 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:27:33.000857 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 05:27:33.000991 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":338,"threads_started":1}
I20260928 05:27:33.001787 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.995630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:48129 (local address 127.8.124.195:46111)
0928 05:27:32.995807 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:32.995812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:32.996370 (+ 558us) server_negotiation.cc:407] Connection header received
0928 05:27:32.996545 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:32.996549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:32.996606 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:32.996666 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:32.997188 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.998136 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.999933 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.000832 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.000971 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.001034 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.001397 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.001528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:27:33.001834 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:32.995534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:46111 (local address 127.8.124.194:48129)
0928 05:27:32.996267 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:32.996281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:32.996409 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:32.996699 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:32.996703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:32.996728 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:32.997066 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.997077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:32.998262 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:32.998266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:32.999767 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:32.999782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.001037 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.001043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:33.001226 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.001262 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:27:33.001359 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:27:33.001475 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":427,"threads_started":1}
I20260928 05:27:33.002784 26228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7382fb5fe9624207a67adf76d01008e1" candidate_uuid: "81df71e931b74824a9610f23a7bdc127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" is_pre_election: true
I20260928 05:27:33.002784 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7382fb5fe9624207a67adf76d01008e1" candidate_uuid: "81df71e931b74824a9610f23a7bdc127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" is_pre_election: true
I20260928 05:27:33.003360 25961 raft_consensus.cc:2454] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81df71e931b74824a9610f23a7bdc127 in term 0.
I20260928 05:27:33.003360 26228 raft_consensus.cc:2454] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81df71e931b74824a9610f23a7bdc127 in term 0.
I20260928 05:27:33.003988 26030 leader_election.cc:302] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8a294b40c449deb9520f5c9fe0b86c, 81df71e931b74824a9610f23a7bdc127; no voters:
I20260928 05:27:33.004338 26297 raft_consensus.cc:2781] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:33.004478 26297 raft_consensus.cc:482] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:33.004580 26297 raft_consensus.cc:3037] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:33.006763 26297 raft_consensus.cc:504] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:33.007315 26297 leader_election.cc:288] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8a294b40c449deb9520f5c9fe0b86c (127.8.124.193:37357), 1d01f94ac9a94d2b92e7c4c8788d83d8 (127.8.124.195:46111)
I20260928 05:27:33.007766 25961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7382fb5fe9624207a67adf76d01008e1" candidate_uuid: "81df71e931b74824a9610f23a7bdc127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8a294b40c449deb9520f5c9fe0b86c"
I20260928 05:27:33.007819 26228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7382fb5fe9624207a67adf76d01008e1" candidate_uuid: "81df71e931b74824a9610f23a7bdc127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8"
I20260928 05:27:33.007977 25961 raft_consensus.cc:3037] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:33.007995 26228 raft_consensus.cc:3037] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:33.011116 25961 raft_consensus.cc:2454] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81df71e931b74824a9610f23a7bdc127 in term 1.
I20260928 05:27:33.011116 26228 raft_consensus.cc:2454] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81df71e931b74824a9610f23a7bdc127 in term 1.
I20260928 05:27:33.011523 26030 leader_election.cc:302] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8a294b40c449deb9520f5c9fe0b86c, 81df71e931b74824a9610f23a7bdc127; no voters:
I20260928 05:27:33.011783 26297 raft_consensus.cc:2781] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:33.012229 26297 raft_consensus.cc:686] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 1 LEADER]: Becoming Leader. State: Replica: 81df71e931b74824a9610f23a7bdc127, State: Running, Role: LEADER
I20260928 05:27:33.012667 26297 consensus_queue.cc:237] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } }
I20260928 05:27:33.017259 25817 catalog_manager.cc:5495] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81df71e931b74824a9610f23a7bdc127 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "81df71e931b74824a9610f23a7bdc127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81df71e931b74824a9610f23a7bdc127" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 } health_report { overall_health: UNKNOWN } } }
W20260928 05:27:33.023471 26141 tablet.cc:2369] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:27:33.049988 26007 tablet.cc:2369] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:33.071333 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:33.066490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.8.124.194:38207)
0928 05:27:33.066669 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:33.066673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:33.066691 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:27:33.066835 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:33.066839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:33.066917 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:33.067031 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:33.067320 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.068052 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.069358 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.069807 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.069908 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.070343 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:33.070849 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:33.070963 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.070999 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.071063 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:27:33.079280 26228 raft_consensus.cc:1261] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 1 FOLLOWER]: Refusing update from remote peer 81df71e931b74824a9610f23a7bdc127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:33.079309 25961 raft_consensus.cc:1261] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 1 FOLLOWER]: Refusing update from remote peer 81df71e931b74824a9610f23a7bdc127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:33.080165 26302 consensus_queue.cc:1035] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8a294b40c449deb9520f5c9fe0b86c" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:33.080494 26297 consensus_queue.cc:1035] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d01f94ac9a94d2b92e7c4c8788d83d8" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 46111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:33.104532 25817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:33.104792 25817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:33.108279 25817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83941ffe617c4b089b0d893c95d0439c: Sending DeleteTablet for 3 replicas of tablet 7382fb5fe9624207a67adf76d01008e1
I20260928 05:27:33.109370 26075 tablet_service.cc:1514] Processing DeleteTablet for tablet 7382fb5fe9624207a67adf76d01008e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:33 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:47724
I20260928 05:27:33.109484 26208 tablet_service.cc:1514] Processing DeleteTablet for tablet 7382fb5fe9624207a67adf76d01008e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:33 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:57880
I20260928 05:27:33.109484 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 7382fb5fe9624207a67adf76d01008e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:33 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:52478
I20260928 05:27:33.110563 26315 tablet_replica.cc:330] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: stopping tablet replica
I20260928 05:27:33.111034 26315 raft_consensus.cc:2229] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:33.111732 26315 raft_consensus.cc:2258] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:33.112797 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25878
I20260928 05:27:33.117431 26315 ts_tablet_manager.cc:1905] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:33.129889 26317 tablet_replica.cc:330] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: stopping tablet replica
I20260928 05:27:33.130605 26317 raft_consensus.cc:2229] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:33.131273 26317 raft_consensus.cc:2258] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:33.132151 26316 tablet_replica.cc:330] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: stopping tablet replica
I20260928 05:27:33.132550 26316 raft_consensus.cc:2229] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:33.133083 26316 raft_consensus.cc:2258] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 05:27:33 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:27:33.135777 26316 ts_tablet_manager.cc:1905] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:33.135845 26317 ts_tablet_manager.cc:1905] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:33.143501 26316 ts_tablet_manager.cc:1918] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:33.143513 26315 ts_tablet_manager.cc:1918] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:33.143698 26315 log.cc:1196] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/7382fb5fe9624207a67adf76d01008e1
I20260928 05:27:33.143844 26316 log.cc:1196] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/7382fb5fe9624207a67adf76d01008e1
I20260928 05:27:33.144275 26315 ts_tablet_manager.cc:1939] T 7382fb5fe9624207a67adf76d01008e1 P 81df71e931b74824a9610f23a7bdc127: Deleting consensus metadata
I20260928 05:27:33.144395 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:33.113909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.8.124.193:37357)
0928 05:27:33.114085 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:33.114091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:33.124921 (+ 10830us) server_negotiation.cc:407] Connection header received
0928 05:27:33.125548 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:33.125555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:33.125668 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:33.125749 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:33.127598 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.128767 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.130599 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.131241 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.131359 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.136842 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:33.136861 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:33.136878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:33.136923 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:33.140772 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.141714 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.141724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.141727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.141904 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.142393 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.142398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.142401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.142821 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:33.143106 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.143736 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.143981 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:27:33.144914 26316 ts_tablet_manager.cc:1939] T 7382fb5fe9624207a67adf76d01008e1 P 7a8a294b40c449deb9520f5c9fe0b86c: Deleting consensus metadata
I20260928 05:27:33.145882 26001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:33.146246 25804 catalog_manager.cc:4841] TS 81df71e931b74824a9610f23a7bdc127 (127.8.124.194:38207): tablet 7382fb5fe9624207a67adf76d01008e1 (table test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]) successfully deleted
I20260928 05:27:33.147967 26317 ts_tablet_manager.cc:1918] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:33.148149 26317 log.cc:1196] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/7382fb5fe9624207a67adf76d01008e1
I20260928 05:27:33.148662 26317 ts_tablet_manager.cc:1939] T 7382fb5fe9624207a67adf76d01008e1 P 1d01f94ac9a94d2b92e7c4c8788d83d8: Deleting consensus metadata
I20260928 05:27:33.150563 25803 catalog_manager.cc:4841] TS 1d01f94ac9a94d2b92e7c4c8788d83d8 (127.8.124.195:46111): tablet 7382fb5fe9624207a67adf76d01008e1 (table test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]) successfully deleted
I20260928 05:27:33.254832 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25878
I20260928 05:27:33.254992 25803 catalog_manager.cc:4841] TS 7a8a294b40c449deb9520f5c9fe0b86c (127.8.124.193:37357): tablet 7382fb5fe9624207a67adf76d01008e1 (table test-table [id=1594aca4d5bf44eb95fa83262c83d6fc]) successfully deleted
I20260928 05:27:33.275668 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26012
Sep 28 05:27:33 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:27:33.295284 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:33.276627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.8.124.194:38207)
0928 05:27:33.276809 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:33.276816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:33.276953 (+ 137us) server_negotiation.cc:407] Connection header received
0928 05:27:33.277135 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:33.277141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:33.277239 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:33.277366 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:33.278928 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.280203 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.281974 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.282651 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.282776 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.288122 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:33.288136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:33.288155 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:33.288203 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:33.292230 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.293094 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.293104 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.293108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.293298 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.293811 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.293815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.293818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.294181 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:33.294449 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.294791 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.294975 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:27:33.296001 26135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:33.423286 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26012
I20260928 05:27:33.438501 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26145
Sep 28 05:27:33 dist-test-slave-515x krb5kdc[25772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:27:33.452389 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:33.439295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.8.124.195:46111)
0928 05:27:33.439462 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:33.439466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:33.439484 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:27:33.439589 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:33.439592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:33.439656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:33.439738 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:33.440927 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.441652 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.442896 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:33.443265 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:33.443348 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:33.447506 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:33.447521 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:33.447537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:33.447575 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:33.450102 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.450699 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.450705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.450708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.450852 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:33.451126 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:33.451129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:33.451131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:33.451468 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:33.451678 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:33.451925 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 05:27:33.452079 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:27:33.452997 26268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:33.540010 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26145
I20260928 05:27:33.553218 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 25785
I20260928 05:27:33.554436 25847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:33.663859 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 25785
2026-09-28T05:27:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4149 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T05:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:33Z Disabled control of system clock
I20260928 05:27:33.731576 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37971
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:34.046434 26326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:34.046706 26326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:34.046798 26326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:34.053978 26326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:34.054082 26326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:34.054118 26326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:34.054174 26326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:34.054219 26326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:34.063298 26326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:37971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:34.064515 26326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:34.066265 26326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:34.077570 26331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:34.077544 26332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:34.077527 26334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:34.078619 26326 server_base.cc:1034] running on GCE node
I20260928 05:27:34.079433 26326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:34.080878 26326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:34.082085 26326 hybrid_clock.cc:648] HybridClock initialized: now 1790573254082009 us; error 63 us; skew 500 ppm
I20260928 05:27:34.082433 26326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:34.084658 26326 webserver.cc:460] Webserver started at http://127.8.124.254:37069/ using document root <none> and password file <none>
I20260928 05:27:34.085316 26326 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:34.085414 26326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:34.085759 26326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:34.088567 26326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "015d93d30739417a877f3af50c4a8557"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "8d96b29249cdcb99100d7888002b8a4e"
server_key_iv: "0aeee752e02566de404a310e438d5a93"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.089401 26326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "015d93d30739417a877f3af50c4a8557"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "8d96b29249cdcb99100d7888002b8a4e"
server_key_iv: "0aeee752e02566de404a310e438d5a93"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.094451 26326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 05:27:34.097940 26340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:34.099349 26326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:27:34.099535 26326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "015d93d30739417a877f3af50c4a8557"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "8d96b29249cdcb99100d7888002b8a4e"
server_key_iv: "0aeee752e02566de404a310e438d5a93"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.099745 26326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:34.134963 26326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:34.135833 26326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:34.136094 26326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:34.153672 26326 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:37971
I20260928 05:27:34.153729 26391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:37971 every 8 connection(s)
I20260928 05:27:34.154992 26326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:34.158648 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:34.162388 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26326
I20260928 05:27:34.162565 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:34.163122 8691 external_mini_cluster.cc:1428] Setting key a7bc98b863e7e1b33a2752a22a01a064
I20260928 05:27:34.170961 26392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: Bootstrap starting.
I20260928 05:27:34.173938 26392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:34.174954 26392 log.cc:824] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:34.175989 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:34.164988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.8.124.254:37971)
0928 05:27:34.165972 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:34.165983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:34.166024 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:27:34.167597 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:34.167619 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:34.168090 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:34.168457 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:34.169012 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.170338 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.171764 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.172450 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.172617 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:34.173722 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:34.173730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:34.173744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:34.174316 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:34.174410 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:34.174430 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:34.174524 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:34.174744 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 05:27:34.175032 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260928 05:27:34.177843 26392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: No bootstrap required, opened a new log
I20260928 05:27:34.185637 26392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } }
I20260928 05:27:34.185976 26392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:34.186049 26392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015d93d30739417a877f3af50c4a8557, State: Initialized, Role: FOLLOWER
I20260928 05:27:34.186543 26392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } }
I20260928 05:27:34.186758 26392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:34.186852 26392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:34.186966 26392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:34.188942 26392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } }
I20260928 05:27:34.189360 26392 leader_election.cc:302] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015d93d30739417a877f3af50c4a8557; no voters:
I20260928 05:27:34.189870 26392 leader_election.cc:288] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:34.189983 26397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:34.190740 26397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [term 1 LEADER]: Becoming Leader. State: Replica: 015d93d30739417a877f3af50c4a8557, State: Running, Role: LEADER
I20260928 05:27:34.191212 26397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } }
I20260928 05:27:34.191756 26392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:34.193584 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015d93d30739417a877f3af50c4a8557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } } }
I20260928 05:27:34.194074 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:34.193958 26399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015d93d30739417a877f3af50c4a8557. Latest consensus state: current_term: 1 leader_uuid: "015d93d30739417a877f3af50c4a8557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015d93d30739417a877f3af50c4a8557" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37971 } } }
I20260928 05:27:34.194344 26399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:34.197839 26403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:34.202569 26403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:34.210541 26403 catalog_manager.cc:1347] Generated new cluster ID: 4f167e4ca66a429b8e23dc5a81039176
I20260928 05:27:34.210702 26403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:34.223277 26403 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:34.224596 26403 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:34.234418 26403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: Generated new TSK 0
I20260928 05:27:34.235291 26403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:34.241444 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--builtin_ntp_servers=127.8.124.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:34.563364 26416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:34.563657 26416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:34.563791 26416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:34.571053 26416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:34.571166 26416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:34.571305 26416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:34.580387 26416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:34.581691 26416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:34.583472 26416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:34.595965 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:34.596057 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:34.596211 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:34.597147 26416 server_base.cc:1034] running on GCE node
I20260928 05:27:34.597873 26416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:34.598832 26416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:34.600031 26416 hybrid_clock.cc:648] HybridClock initialized: now 1790573254599973 us; error 53 us; skew 500 ppm
I20260928 05:27:34.600335 26416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:34.605958 26416 webserver.cc:460] Webserver started at http://127.8.124.193:40329/ using document root <none> and password file <none>
I20260928 05:27:34.606681 26416 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:34.606789 26416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:34.607093 26416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:34.610008 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "d572a27022fa4ef4bf7897ed3424d289"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "3b751c20584aef3f1772b3d56dafa1a5"
server_key_iv: "afff029283f44bef7f8508d8d51275f3"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.610786 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "d572a27022fa4ef4bf7897ed3424d289"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "3b751c20584aef3f1772b3d56dafa1a5"
server_key_iv: "afff029283f44bef7f8508d8d51275f3"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.615903 26416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 05:27:34.619669 26430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:34.621060 26416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:27:34.621246 26416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "d572a27022fa4ef4bf7897ed3424d289"
format_stamp: "Formatted at 2026-09-28 05:27:34 on dist-test-slave-515x"
server_key: "3b751c20584aef3f1772b3d56dafa1a5"
server_key_iv: "afff029283f44bef7f8508d8d51275f3"
server_key_version: "encryptionkey@0"
I20260928 05:27:34.621456 26416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:34.657070 26416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:34.657967 26416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:34.658260 26416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:34.659226 26416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:34.660646 26416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:34.660744 26416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:34.660849 26416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:34.660915 26416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:34.690052 26416 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:36111
I20260928 05:27:34.690126 26542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:36111 every 8 connection(s)
I20260928 05:27:34.691311 26416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:34.694464 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26416
I20260928 05:27:34.694640 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:34.695228 8691 external_mini_cluster.cc:1428] Setting key 115f360a7260c5153d5899ff47858b8f
I20260928 05:27:34.698022 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--builtin_ntp_servers=127.8.124.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:34.704416 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:34.693774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:59895 (local address 127.8.124.254:37971)
0928 05:27:34.694220 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:34.694225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:34.695634 (+ 1409us) server_negotiation.cc:407] Connection header received
0928 05:27:34.697816 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:34.697822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:34.697899 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:34.697985 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:34.699116 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.699792 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.701559 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.702190 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.702319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:34.703655 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:34.703661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:34.703665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:34.703685 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:34.703732 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:34.703747 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:34.703828 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:34.703992 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 05:27:34.704087 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":217,"threads_started":1}
I20260928 05:27:34.705423 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:34.694393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37971 (local address 127.8.124.193:59895)
0928 05:27:34.695461 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:34.695507 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:34.697545 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:34.698223 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:34.698235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:34.698339 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:34.698966 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:34.698983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.699930 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.699936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:34.701412 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:34.701425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:34.702392 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:34.702397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:34.702639 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:34.703341 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:34.703344 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:34.703386 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:34.703470 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:34.703829 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:34.703835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:34.703841 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:27:34.703981 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 05:27:34.704321 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":360,"threads_started":1}
I20260928 05:27:34.706823 26543 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37971
I20260928 05:27:34.707103 26543 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:34.707911 26543 heartbeater.cc:511] Master 127.8.124.254:37971 requested a full tablet report, sending...
I20260928 05:27:34.709892 26357 ts_manager.cc:194] Registered new tserver with Master: d572a27022fa4ef4bf7897ed3424d289 (127.8.124.193:36111)
I20260928 05:27:34.711345 26357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.193:59895
W20260928 05:27:35.010629 26547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:35.010934 26547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:35.011065 26547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:35.018226 26547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:35.018328 26547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:35.018463 26547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:35.027678 26547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:35.028904 26547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:35.030715 26547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:35.042831 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:35.042891 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:35.042842 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:35.043560 26547 server_base.cc:1034] running on GCE node
I20260928 05:27:35.044128 26547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:35.045028 26547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:35.046218 26547 hybrid_clock.cc:648] HybridClock initialized: now 1790573255046164 us; error 50 us; skew 500 ppm
I20260928 05:27:35.046516 26547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:35.048695 26547 webserver.cc:460] Webserver started at http://127.8.124.194:46643/ using document root <none> and password file <none>
I20260928 05:27:35.049346 26547 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:35.049443 26547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:35.049777 26547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:35.052510 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "67010f506b82425aa6ee72b19b918940"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "b21d0d97d3316e8269368bf98c35cd73"
server_key_iv: "8efe5938a7233684b8683d76201bef8c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.053227 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "67010f506b82425aa6ee72b19b918940"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "b21d0d97d3316e8269368bf98c35cd73"
server_key_iv: "8efe5938a7233684b8683d76201bef8c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.057978 26547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260928 05:27:35.061182 26561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.062546 26547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:35.062721 26547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "67010f506b82425aa6ee72b19b918940"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "b21d0d97d3316e8269368bf98c35cd73"
server_key_iv: "8efe5938a7233684b8683d76201bef8c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.062920 26547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:35.086046 26547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:35.086869 26547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:35.087154 26547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:35.088099 26547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:35.089540 26547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:35.089659 26547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.089757 26547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:35.089831 26547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.122831 26547 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:42251
I20260928 05:27:35.122902 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:42251 every 8 connection(s)
I20260928 05:27:35.124137 26547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:35.132177 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26547
I20260928 05:27:35.132336 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:35.132851 8691 external_mini_cluster.cc:1428] Setting key 983727bdf91b44a8431ca1d3a61fe759
I20260928 05:27:35.136217 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--builtin_ntp_servers=127.8.124.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:35.150158 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.128314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:51609 (local address 127.8.124.254:37971)
0928 05:27:35.128487 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.128493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.140640 (+ 12147us) server_negotiation.cc:407] Connection header received
0928 05:27:35.142787 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.142794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.142880 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.142988 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:35.144459 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.145176 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.147392 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.147788 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.147887 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.149402 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:35.149407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:35.149411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:35.149432 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:35.149465 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:35.149477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:35.149554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.149821 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.149913 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:27:35.151568 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.127733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37971 (local address 127.8.124.194:51609)
0928 05:27:35.140457 (+ 12724us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.140506 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.142485 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.143261 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.143276 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.143419 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:35.144260 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.144288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.145359 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.145364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.147237 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.147253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.148003 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.148009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.148288 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.149031 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:35.149035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:35.149087 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:35.149196 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:35.149567 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:35.149576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:35.149583 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.149808 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.150237 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12103,"thread_start_us":13776,"threads_started":1}
I20260928 05:27:35.153189 26674 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37971
I20260928 05:27:35.153519 26674 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:35.154465 26674 heartbeater.cc:511] Master 127.8.124.254:37971 requested a full tablet report, sending...
I20260928 05:27:35.156164 26357 ts_manager.cc:194] Registered new tserver with Master: 67010f506b82425aa6ee72b19b918940 (127.8.124.194:42251)
I20260928 05:27:35.157377 26357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.194:51609
W20260928 05:27:35.455302 26677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:35.455575 26677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:35.455657 26677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:35.462837 26677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:35.462937 26677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:35.463042 26677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:35.472189 26677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:35.473429 26677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:35.475138 26677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:35.487793 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:35.487798 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:35.487936 26685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:35.488581 26677 server_base.cc:1034] running on GCE node
I20260928 05:27:35.489248 26677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:35.490379 26677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:35.491580 26677 hybrid_clock.cc:648] HybridClock initialized: now 1790573255491528 us; error 47 us; skew 500 ppm
I20260928 05:27:35.491904 26677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:35.494181 26677 webserver.cc:460] Webserver started at http://127.8.124.195:45391/ using document root <none> and password file <none>
I20260928 05:27:35.494848 26677 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:35.494939 26677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:35.495234 26677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:35.498067 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "933a4d977e78430aa05cabd7d2dc8381"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "cdf71b60a35f2a8b74f96ad3e1cbb1ca"
server_key_iv: "66a6dc4da7458c765b5b6611e0ddc07c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.498843 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "933a4d977e78430aa05cabd7d2dc8381"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "cdf71b60a35f2a8b74f96ad3e1cbb1ca"
server_key_iv: "66a6dc4da7458c765b5b6611e0ddc07c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.503945 26677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:27:35.507378 26691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.508633 26677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:27:35.508787 26677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "933a4d977e78430aa05cabd7d2dc8381"
format_stamp: "Formatted at 2026-09-28 05:27:35 on dist-test-slave-515x"
server_key: "cdf71b60a35f2a8b74f96ad3e1cbb1ca"
server_key_iv: "66a6dc4da7458c765b5b6611e0ddc07c"
server_key_version: "encryptionkey@0"
I20260928 05:27:35.508982 26677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:35.528874 26677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:35.529722 26677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:35.529995 26677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:35.530930 26677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:35.532302 26677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:35.532390 26677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.532493 26677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:35.532560 26677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:35.561813 26677 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:33815
I20260928 05:27:35.561872 26803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:33815 every 8 connection(s)
I20260928 05:27:35.563058 26677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:35.572300 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26677
I20260928 05:27:35.572448 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:35.572877 8691 external_mini_cluster.cc:1428] Setting key e7dd314a897500a15ed340f9cbe19be0
I20260928 05:27:35.575565 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.565568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:55533 (local address 127.8.124.254:37971)
0928 05:27:35.565808 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.565813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.567290 (+ 1477us) server_negotiation.cc:407] Connection header received
0928 05:27:35.569073 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.569080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.569165 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.569252 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:35.570349 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.571095 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.572662 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.573100 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.573211 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.574778 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:35.574785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:35.574789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:35.574812 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:35.574853 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:35.574868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:35.574954 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.575148 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.575247 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 05:27:35.576506 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.566123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37971 (local address 127.8.124.195:55533)
0928 05:27:35.567072 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.567147 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.568802 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.569483 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.569492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.569581 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:35.570189 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.570206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.571249 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.571253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.572525 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.572540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.573321 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.573329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.573646 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.574390 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:35.574393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:35.574434 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:35.574545 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:35.574952 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:35.574960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:35.574965 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.575095 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.575400 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":272,"threads_started":1}
I20260928 05:27:35.577852 26804 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37971
I20260928 05:27:35.578133 26804 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:35.578860 26804 heartbeater.cc:511] Master 127.8.124.254:37971 requested a full tablet report, sending...
I20260928 05:27:35.580262 26357 ts_manager.cc:194] Registered new tserver with Master: 933a4d977e78430aa05cabd7d2dc8381 (127.8.124.195:33815)
I20260928 05:27:35.581058 26357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.195:55533
I20260928 05:27:35.588030 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:35.597944 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.592080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.8.124.254:37971)
0928 05:27:35.592248 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.592254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.592643 (+ 389us) server_negotiation.cc:407] Connection header received
0928 05:27:35.592811 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.592818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.592916 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.593013 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:35.593528 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.594279 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.595645 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.596141 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.596249 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.597247 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:35.597254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:35.597258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:35.597276 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:35.597310 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:35.597322 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:35.597393 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.597519 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.597640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:27:35.601575 26357 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:35.608423 26357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:35.643788 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.620551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:42251 (local address 127.0.0.1:33928)
0928 05:27:35.629348 (+ 8797us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.629407 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.629638 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.635044 (+ 5406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.635052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.635091 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:35.635501 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.635514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.637764 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.637770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.640557 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.640574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.642852 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.642859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.643085 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.643153 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.643262 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.643452 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8473,"threads_started":1}
I20260928 05:27:35.645345 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.8.124.194:42251)
0928 05:27:35.633677 (+ 12930us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.633687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.633723 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:27:35.633840 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.633848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.634122 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.635741 (+ 1619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:35.636012 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.637579 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.641106 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.642628 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.644420 (+ 1792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.644723 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.644794 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.644968 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12728,"thread_start_us":13933,"threads_started":1}
I20260928 05:27:35.652585 26609 tablet_service.cc:1467] Processing CreateTablet for tablet 521342b365d149e1bc834c13bbb71cbb (DEFAULT_TABLE table=test-table [id=98618b8671514556b0973252e5bac6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:35.654230 26609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521342b365d149e1bc834c13bbb71cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:35.654349 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.620984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:33815 (local address 127.0.0.1:56992)
0928 05:27:35.646920 (+ 25936us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.646937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.647050 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.647975 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.647980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.648003 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:35.648378 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.648391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.650180 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.650185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.652215 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.652230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.653527 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.653534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.653729 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.653763 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.653853 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.653974 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25819,"mutex_wait_us":22,"thread_start_us":7889,"threads_started":1}
I20260928 05:27:35.654378 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.622000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.8.124.195:33815)
0928 05:27:35.646167 (+ 24167us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.646177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.647174 (+ 997us) server_negotiation.cc:407] Connection header received
0928 05:27:35.647275 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.647282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.647510 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.647724 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:35.648569 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.650036 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.652406 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.653345 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.653472 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.653767 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.653892 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.654049 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23983,"thread_start_us":23827,"threads_started":1}
I20260928 05:27:35.655469 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.620647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45910)
0928 05:27:35.644147 (+ 23500us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.644169 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.644270 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.647219 (+ 2949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.647223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.647247 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:35.647407 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.647417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.649333 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.649338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.651898 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.651913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.654745 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.654751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.654926 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.654957 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.655042 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.655174 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23368,"mutex_wait_us":98,"thread_start_us":26033,"threads_started":1}
I20260928 05:27:35.655548 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.621643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.8.124.193:36111)
0928 05:27:35.646572 (+ 24929us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.646580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.646613 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:27:35.646715 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.646724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.647003 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.647578 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:35.647846 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.649195 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.652074 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.654594 (+ 2520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.654723 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.654931 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.655082 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.655248 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24788,"thread_start_us":24691,"threads_started":1}
I20260928 05:27:35.658056 26739 tablet_service.cc:1467] Processing CreateTablet for tablet 521342b365d149e1bc834c13bbb71cbb (DEFAULT_TABLE table=test-table [id=98618b8671514556b0973252e5bac6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:35.659479 26739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521342b365d149e1bc834c13bbb71cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:35.659037 26478 tablet_service.cc:1467] Processing CreateTablet for tablet 521342b365d149e1bc834c13bbb71cbb (DEFAULT_TABLE table=test-table [id=98618b8671514556b0973252e5bac6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:35.660364 26478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521342b365d149e1bc834c13bbb71cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:35.666262 26822 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Bootstrap starting.
I20260928 05:27:35.669842 26822 tablet_bootstrap.cc:654] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:35.670769 26822 log.cc:824] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:35.673220 26822 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: No bootstrap required, opened a new log
I20260928 05:27:35.673574 26822 ts_tablet_manager.cc:1397] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:27:35.673576 26824 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Bootstrap starting.
I20260928 05:27:35.675091 26825 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: Bootstrap starting.
I20260928 05:27:35.677716 26824 tablet_bootstrap.cc:654] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:35.677845 26825 tablet_bootstrap.cc:654] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:35.678738 26825 log.cc:824] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:35.678839 26824 log.cc:824] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:35.681326 26824 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: No bootstrap required, opened a new log
I20260928 05:27:35.681553 26824 ts_tablet_manager.cc:1397] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 05:27:35.681558 26825 tablet_bootstrap.cc:492] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: No bootstrap required, opened a new log
I20260928 05:27:35.681830 26825 ts_tablet_manager.cc:1397] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260928 05:27:35.684108 26822 raft_consensus.cc:348] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.685035 26822 raft_consensus.cc:374] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:35.685297 26822 raft_consensus.cc:729] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67010f506b82425aa6ee72b19b918940, State: Initialized, Role: FOLLOWER
I20260928 05:27:35.685979 26822 consensus_queue.cc:260] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.687131 26674 heartbeater.cc:503] Master 127.8.124.254:37971 was elected leader, sending a full tablet report...
I20260928 05:27:35.689113 26822 ts_tablet_manager.cc:1428] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 05:27:35.689594 26824 raft_consensus.cc:348] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.690100 26824 raft_consensus.cc:374] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:35.690244 26824 raft_consensus.cc:729] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933a4d977e78430aa05cabd7d2dc8381, State: Initialized, Role: FOLLOWER
I20260928 05:27:35.691000 26824 consensus_queue.cc:260] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.691993 26804 heartbeater.cc:503] Master 127.8.124.254:37971 was elected leader, sending a full tablet report...
I20260928 05:27:35.692632 26824 ts_tablet_manager.cc:1428] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260928 05:27:35.694022 26825 raft_consensus.cc:348] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.694464 26825 raft_consensus.cc:374] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:35.694555 26825 raft_consensus.cc:729] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d572a27022fa4ef4bf7897ed3424d289, State: Initialized, Role: FOLLOWER
I20260928 05:27:35.695169 26825 consensus_queue.cc:260] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.698279 26543 heartbeater.cc:503] Master 127.8.124.254:37971 was elected leader, sending a full tablet report...
I20260928 05:27:35.698886 26825 ts_tablet_manager.cc:1428] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260928 05:27:35.748381 26829 raft_consensus.cc:482] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:35.748672 26829 raft_consensus.cc:504] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.750423 26829 leader_election.cc:288] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67010f506b82425aa6ee72b19b918940 (127.8.124.194:42251), d572a27022fa4ef4bf7897ed3424d289 (127.8.124.193:36111)
I20260928 05:27:35.756558 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.750646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:42251 (local address 127.8.124.195:55527)
0928 05:27:35.750931 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.750953 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.751074 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.751481 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.751486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.751512 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:35.751692 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.751704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.752918 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.752924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.754555 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.754567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.755809 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.755815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.755980 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.756016 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.756110 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.756242 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":47}
I20260928 05:27:35.756605 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.750771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:55527 (local address 127.8.124.194:42251)
0928 05:27:35.750993 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.750998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.751018 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:27:35.751262 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.751266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.751336 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.751410 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:35.751848 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.752778 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.754709 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.755657 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.755774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.755824 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.756236 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.756366 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 05:27:35.757085 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.751020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:43793 (local address 127.8.124.193:36111)
0928 05:27:35.751206 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.751211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.751650 (+ 439us) server_negotiation.cc:407] Connection header received
0928 05:27:35.751784 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.751789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.751877 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.751943 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:35.752512 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.753354 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.755287 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.756215 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.756359 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.756423 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.756795 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.756907 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:27:35.757625 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.750854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.8.124.195:43793)
0928 05:27:35.751547 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:35.751564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:35.751670 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:35.751995 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:35.751999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:35.752023 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:35.752361 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.752372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.753518 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.753523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.755106 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:35.755119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.756382 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.756389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:35.756572 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.756604 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:27:35.757253 (+ 649us) client_negotiation.cc:240] Negotiation successful
0928 05:27:35.757368 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":1,"thread_start_us":371,"threads_started":1}
I20260928 05:27:35.757568 26629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521342b365d149e1bc834c13bbb71cbb" candidate_uuid: "933a4d977e78430aa05cabd7d2dc8381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67010f506b82425aa6ee72b19b918940" is_pre_election: true
I20260928 05:27:35.758116 26629 raft_consensus.cc:2454] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933a4d977e78430aa05cabd7d2dc8381 in term 0.
I20260928 05:27:35.758533 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521342b365d149e1bc834c13bbb71cbb" candidate_uuid: "933a4d977e78430aa05cabd7d2dc8381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d572a27022fa4ef4bf7897ed3424d289" is_pre_election: true
I20260928 05:27:35.758737 26694 leader_election.cc:302] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67010f506b82425aa6ee72b19b918940, 933a4d977e78430aa05cabd7d2dc8381; no voters:
I20260928 05:27:35.759126 26829 raft_consensus.cc:2781] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:35.759076 26498 raft_consensus.cc:2454] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933a4d977e78430aa05cabd7d2dc8381 in term 0.
I20260928 05:27:35.759248 26829 raft_consensus.cc:482] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:35.759354 26829 raft_consensus.cc:3037] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:35.761514 26829 raft_consensus.cc:504] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.762084 26829 leader_election.cc:288] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [CANDIDATE]: Term 1 election: Requested vote from peers 67010f506b82425aa6ee72b19b918940 (127.8.124.194:42251), d572a27022fa4ef4bf7897ed3424d289 (127.8.124.193:36111)
I20260928 05:27:35.762512 26629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521342b365d149e1bc834c13bbb71cbb" candidate_uuid: "933a4d977e78430aa05cabd7d2dc8381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67010f506b82425aa6ee72b19b918940"
I20260928 05:27:35.762569 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521342b365d149e1bc834c13bbb71cbb" candidate_uuid: "933a4d977e78430aa05cabd7d2dc8381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d572a27022fa4ef4bf7897ed3424d289"
I20260928 05:27:35.762673 26629 raft_consensus.cc:3037] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:35.762759 26498 raft_consensus.cc:3037] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:35.764786 26629 raft_consensus.cc:2454] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933a4d977e78430aa05cabd7d2dc8381 in term 1.
I20260928 05:27:35.764886 26498 raft_consensus.cc:2454] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933a4d977e78430aa05cabd7d2dc8381 in term 1.
I20260928 05:27:35.765153 26694 leader_election.cc:302] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67010f506b82425aa6ee72b19b918940, 933a4d977e78430aa05cabd7d2dc8381; no voters:
I20260928 05:27:35.765415 26829 raft_consensus.cc:2781] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:35.765902 26829 raft_consensus.cc:686] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 1 LEADER]: Becoming Leader. State: Replica: 933a4d977e78430aa05cabd7d2dc8381, State: Running, Role: LEADER
I20260928 05:27:35.766381 26829 consensus_queue.cc:237] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } }
I20260928 05:27:35.770632 26356 catalog_manager.cc:5495] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 reported cstate change: term changed from 0 to 1, leader changed from <none> to 933a4d977e78430aa05cabd7d2dc8381 (127.8.124.195). New cstate: current_term: 1 leader_uuid: "933a4d977e78430aa05cabd7d2dc8381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "933a4d977e78430aa05cabd7d2dc8381" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 33815 } health_report { overall_health: HEALTHY } } }
I20260928 05:27:35.809509 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.804202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.8.124.195:33815)
0928 05:27:35.804411 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.804417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.804529 (+ 112us) server_negotiation.cc:407] Connection header received
0928 05:27:35.804750 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.804754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.804861 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.804983 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:35.805352 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.806089 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.807486 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.807892 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.807997 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.808465 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:35.808967 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:35.809103 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.809200 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.809273 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:27:35.814582 26805 tablet.cc:2369] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:35.817273 26629 raft_consensus.cc:1261] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 1 FOLLOWER]: Refusing update from remote peer 933a4d977e78430aa05cabd7d2dc8381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:35.817302 26498 raft_consensus.cc:1261] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289 [term 1 FOLLOWER]: Refusing update from remote peer 933a4d977e78430aa05cabd7d2dc8381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:35.818172 26832 consensus_queue.cc:1035] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d572a27022fa4ef4bf7897ed3424d289" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:35.818485 26829 consensus_queue.cc:1035] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67010f506b82425aa6ee72b19b918940" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:35.831682 26836 mvcc.cc:205] Tried to move back new op lower bound from 7334188055818797056 to 7334188055623237632. Current Snapshot: MvccSnapshot[applied={T|T < 7334188055818797056 or (T in {7334188055818797056})}]
I20260928 05:27:35.842775 26357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:35.843068 26357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:35.846810 26357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 015d93d30739417a877f3af50c4a8557: Sending DeleteTablet for 3 replicas of tablet 521342b365d149e1bc834c13bbb71cbb
I20260928 05:27:35.847878 26609 tablet_service.cc:1514] Processing DeleteTablet for tablet 521342b365d149e1bc834c13bbb71cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:35 UTC) from {username='slave'} at 127.0.0.1:33928
I20260928 05:27:35.848110 26739 tablet_service.cc:1514] Processing DeleteTablet for tablet 521342b365d149e1bc834c13bbb71cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:35 UTC) from {username='slave'} at 127.0.0.1:56992
I20260928 05:27:35.848616 26478 tablet_service.cc:1514] Processing DeleteTablet for tablet 521342b365d149e1bc834c13bbb71cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:35 UTC) from {username='slave'} at 127.0.0.1:45910
I20260928 05:27:35.849255 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26416
I20260928 05:27:35.868860 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:35.850542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.8.124.193:36111)
0928 05:27:35.850724 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:35.850731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:35.860384 (+ 9653us) server_negotiation.cc:407] Connection header received
0928 05:27:35.860872 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:35.860878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:35.860997 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:35.861085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:35.861788 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.862952 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.864817 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:35.865473 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:35.865567 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:35.867106 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:35.867117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:35.867135 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:35.867966 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:35.868027 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:35.868052 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:35.868133 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:35.868388 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:27:35.868500 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 05:27:35.870262 26538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:35.870913 26845 tablet_replica.cc:330] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: stopping tablet replica
I20260928 05:27:35.871469 26845 raft_consensus.cc:2229] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:35.872104 26845 raft_consensus.cc:2258] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:35.873229 26846 tablet_replica.cc:330] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: stopping tablet replica
I20260928 05:27:35.874517 26845 ts_tablet_manager.cc:1905] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:35.879566 26845 ts_tablet_manager.cc:1918] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:35.879733 26845 log.cc:1196] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/521342b365d149e1bc834c13bbb71cbb
I20260928 05:27:35.879987 26846 raft_consensus.cc:2229] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:35.880265 26845 ts_tablet_manager.cc:1939] T 521342b365d149e1bc834c13bbb71cbb P 67010f506b82425aa6ee72b19b918940: Deleting consensus metadata
I20260928 05:27:35.880955 26846 raft_consensus.cc:2258] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:35.882169 26343 catalog_manager.cc:4841] TS 67010f506b82425aa6ee72b19b918940 (127.8.124.194:42251): tablet 521342b365d149e1bc834c13bbb71cbb (table test-table [id=98618b8671514556b0973252e5bac6cc]) successfully deleted
I20260928 05:27:35.883293 26846 ts_tablet_manager.cc:1905] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:35.888262 26846 ts_tablet_manager.cc:1918] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:35.888437 26846 log.cc:1196] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/521342b365d149e1bc834c13bbb71cbb
I20260928 05:27:35.888899 26846 ts_tablet_manager.cc:1939] T 521342b365d149e1bc834c13bbb71cbb P 933a4d977e78430aa05cabd7d2dc8381: Deleting consensus metadata
I20260928 05:27:35.890410 26341 catalog_manager.cc:4841] TS 933a4d977e78430aa05cabd7d2dc8381 (127.8.124.195:33815): tablet 521342b365d149e1bc834c13bbb71cbb (table test-table [id=98618b8671514556b0973252e5bac6cc]) successfully deleted
I20260928 05:27:36.011057 26847 tablet_replica.cc:330] T 521342b365d149e1bc834c13bbb71cbb P d572a27022fa4ef4bf7897ed3424d289: stopping tablet replica
I20260928 05:27:36.011456 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26416
I20260928 05:27:36.025477 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26547
W20260928 05:27:36.025436 26342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:36111 (error 108)
I20260928 05:27:36.027509 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.026958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45928)
0928 05:27:36.027179 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.027306 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 05:27:36.027843 26342 catalog_manager.cc:4584] TS d572a27022fa4ef4bf7897ed3424d289 (127.8.124.193:36111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 521342b365d149e1bc834c13bbb71cbb: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
I20260928 05:27:36.036484 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.030053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.8.124.194:42251)
0928 05:27:36.030307 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:36.030314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:36.030334 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:27:36.030400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:36.030405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:36.030521 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:36.030629 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:36.030957 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:36.032022 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:36.033258 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:36.033901 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:36.034009 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:36.034964 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:36.034975 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:36.034993 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:36.035739 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:36.035797 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:36.035821 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:36.035923 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:36.036044 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 05:27:36.036147 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:27:36.037160 26669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:36.081331 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.080895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45930)
0928 05:27:36.081102 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.081188 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 05:27:36.082443 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.082030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45934)
0928 05:27:36.082229 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.082304 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:27:36.143080 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.142665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45938)
0928 05:27:36.142856 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.142943 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 05:27:36.144081 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.143746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45946)
0928 05:27:36.143889 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.143954 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 05:27:36.183756 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26547
I20260928 05:27:36.198168 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26677
I20260928 05:27:36.205227 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.198912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.8.124.195:33815)
0928 05:27:36.199078 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:36.199084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:36.199153 (+ 69us) server_negotiation.cc:407] Connection header received
0928 05:27:36.199368 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:36.199372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:36.199458 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:36.199544 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:36.199952 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:36.200726 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:36.202108 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:36.202535 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:36.202643 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:36.203738 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:36.203746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:36.203759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:36.204378 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:36.204450 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:36.204475 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:36.204575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:36.204701 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 05:27:36.204799 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:27:36.206048 26799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:36.224589 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.224164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45952)
0928 05:27:36.224360 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.224445 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:27:36.225733 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:36.225182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36111 (local address 127.0.0.1:45958)
0928 05:27:36.225433 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:36.225525 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 05:27:36.323019 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26677
I20260928 05:27:36.339078 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26326
I20260928 05:27:36.340252 26387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:36.446763 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26326
2026-09-28T05:27:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:36 dist-test-slave-515x krb5kdc[26856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:36 dist-test-slave-515x krb5kdc[26856](info): set up 2 sockets
Sep 28 05:27:36 dist-test-slave-515x krb5kdc[26856](info): commencing operation
krb5kdc: starting...
W20260928 05:27:37.453513 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:37 dist-test-slave-515x krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:27:37.672212 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44653
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:44653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:37.979763 26869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:37.980020 26869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:37.980145 26869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:37.987118 26869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:37.987215 26869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:37.987287 26869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:37.987363 26869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:37.987419 26869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:37.996716 26869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:44653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:37.997946 26869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:37.999146 26869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:38.009745 26874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:38.009819 26877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:38.009771 26875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:38.010581 26869 server_base.cc:1034] running on GCE node
I20260928 05:27:38.011281 26869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:38.012542 26869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:38.013685 26869 hybrid_clock.cc:648] HybridClock initialized: now 1790573258013629 us; error 45 us; skew 500 ppm
I20260928 05:27:38.014008 26869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:38 dist-test-slave-515x krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:38.016826 26869 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:27:38.018177 26869 webserver.cc:460] Webserver started at http://127.8.124.254:36241/ using document root <none> and password file <none>
I20260928 05:27:38.018755 26869 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:38.018857 26869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:38.019095 26869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:38.021894 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "77c401e88e1b4d51b54a5694f7f2491e"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "1ec5a31f1dfcfac45077d3385b6d7b8b"
server_key_iv: "3267a6ce400590afdd098791d940797a"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.022552 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "77c401e88e1b4d51b54a5694f7f2491e"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "1ec5a31f1dfcfac45077d3385b6d7b8b"
server_key_iv: "3267a6ce400590afdd098791d940797a"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.027273 26869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:27:38.030732 26884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:38.031941 26869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:38.032075 26869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "77c401e88e1b4d51b54a5694f7f2491e"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "1ec5a31f1dfcfac45077d3385b6d7b8b"
server_key_iv: "3267a6ce400590afdd098791d940797a"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.032264 26869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:38.050143 26869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:38.053779 26869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:38.053984 26869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:38.070075 26869 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:44653
I20260928 05:27:38.070170 26935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:44653 every 8 connection(s)
I20260928 05:27:38.071403 26869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:38.074961 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:38.081466 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26869
I20260928 05:27:38.081635 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:38.082050 8691 external_mini_cluster.cc:1428] Setting key 34ef893537d6d0ee7a5df912714751a1
I20260928 05:27:38.084975 26936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: Bootstrap starting.
I20260928 05:27:38.088325 26936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:38.089433 26936 log.cc:824] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:38.092631 26936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: No bootstrap required, opened a new log
Sep 28 05:27:38 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:38.101768 26936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } }
I20260928 05:27:38.102226 26936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:38.102334 26936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c401e88e1b4d51b54a5694f7f2491e, State: Initialized, Role: FOLLOWER
I20260928 05:27:38.102972 26936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } }
I20260928 05:27:38.103214 26936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:38.103351 26936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:38.103511 26936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:38.105518 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:38.083758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.8.124.254:44653)
0928 05:27:38.084497 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:38.084508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:38.084546 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:27:38.086272 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:38.086293 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:38.086661 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:38.086971 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:38.088808 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.090398 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.092313 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.093008 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.093147 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:38.098003 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:38.098012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:38.098029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:38.098075 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:38.101184 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:38.102134 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:38.102144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:38.102152 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:38.102349 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:38.102759 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:38.102764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:38.102766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:38.103371 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:38.103661 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:38.104060 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 05:27:38.104465 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260928 05:27:38.106132 26936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } }
I20260928 05:27:38.106616 26936 leader_election.cc:302] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c401e88e1b4d51b54a5694f7f2491e; no voters:
I20260928 05:27:38.107195 26936 leader_election.cc:288] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:38.107345 26941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:38.108700 26941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [term 1 LEADER]: Becoming Leader. State: Replica: 77c401e88e1b4d51b54a5694f7f2491e, State: Running, Role: LEADER
I20260928 05:27:38.109215 26941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } }
I20260928 05:27:38.109332 26936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:38.112659 26943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c401e88e1b4d51b54a5694f7f2491e. Latest consensus state: current_term: 1 leader_uuid: "77c401e88e1b4d51b54a5694f7f2491e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } } }
I20260928 05:27:38.112985 26943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:38.113524 26942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c401e88e1b4d51b54a5694f7f2491e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c401e88e1b4d51b54a5694f7f2491e" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44653 } } }
I20260928 05:27:38.113804 26942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:38.113863 26951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:38.119134 26951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:38.128471 26951 catalog_manager.cc:1347] Generated new cluster ID: 0029b349f4cc4dceb8ca1ed999daa5a1
I20260928 05:27:38.128612 26951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:38.145123 26951 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:38.146936 26951 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:38.156136 26951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: Generated new TSK 0
I20260928 05:27:38.157163 26951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:27:38.218375 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:38.535794 26962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:38.536113 26962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:38.536257 26962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:38.543329 26962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:38.543442 26962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:38.543596 26962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:38.552851 26962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:38.554137 26962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:38.555379 26962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:38.567684 26968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:38.567725 26967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:38.567802 26970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:38.568285 26962 server_base.cc:1034] running on GCE node
I20260928 05:27:38.568850 26962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:38.569741 26962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:38.570940 26962 hybrid_clock.cc:648] HybridClock initialized: now 1790573258570891 us; error 41 us; skew 500 ppm
I20260928 05:27:38.571241 26962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:38 dist-test-slave-515x krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:38.574198 26962 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:27:38.575547 26962 webserver.cc:460] Webserver started at http://127.8.124.193:45885/ using document root <none> and password file <none>
I20260928 05:27:38.576175 26962 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:38.576297 26962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:38.576572 26962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:38.579373 26962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "602e01e037a7416ab16cae161828a6b3"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "6da4c1db556fb9fdc122839d8ca23721"
server_key_iv: "c84f3af42cfd250e515c19fd442cfcd1"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.580120 26962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "602e01e037a7416ab16cae161828a6b3"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "6da4c1db556fb9fdc122839d8ca23721"
server_key_iv: "c84f3af42cfd250e515c19fd442cfcd1"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.585049 26962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:27:38.588380 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:38.589679 26962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:38.589823 26962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "602e01e037a7416ab16cae161828a6b3"
format_stamp: "Formatted at 2026-09-28 05:27:38 on dist-test-slave-515x"
server_key: "6da4c1db556fb9fdc122839d8ca23721"
server_key_iv: "c84f3af42cfd250e515c19fd442cfcd1"
server_key_version: "encryptionkey@0"
I20260928 05:27:38.589993 26962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:38.608615 26962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:38.612521 26962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:38.612835 26962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:38.613873 26962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:38.615236 26962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:38.615315 26962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:38.615432 26962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:38.615481 26962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:38.644330 26962 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:43373
I20260928 05:27:38.644377 27089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:43373 every 8 connection(s)
I20260928 05:27:38.645503 26962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:38.653286 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 26962
I20260928 05:27:38.653456 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:38.653923 8691 external_mini_cluster.cc:1428] Setting key 478eebf17f4593d7eb08a9b7a6881d0b
Sep 28 05:27:38 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:38.665196 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:38.647794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:56263 (local address 127.8.124.254:44653)
0928 05:27:38.648219 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:38.648225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:38.649122 (+ 897us) server_negotiation.cc:407] Connection header received
0928 05:27:38.650925 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:38.650930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:38.650996 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:38.651080 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:38.652465 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.653341 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.655020 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.655667 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.655803 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:38.659103 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:38.659110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:38.659114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:38.659151 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:38.662003 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:38.662775 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:38.662781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:38.662784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:38.662921 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:38.663412 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:38.663417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:38.663420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:38.663751 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:38.663878 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:38.664764 (+ 886us) server_negotiation.cc:299] Negotiation successful
0928 05:27:38.664934 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":193,"threads_started":1}
I20260928 05:27:38.666569 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:38.648327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:44653 (local address 127.8.124.193:56263)
0928 05:27:38.648964 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:38.649006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:38.650678 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:38.651270 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:38.651277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:38.651773 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:38.652313 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:38.652331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.653494 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.653498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:38.654896 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:38.654908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:38.655910 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:38.655917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:38.656119 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:38.656837 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:38.656864 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:38.658892 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:38.662227 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:38.662234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:38.662247 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:38.662654 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:38.663056 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:38.663061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:38.663064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:38.663305 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:38.663904 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:38.663913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:38.664394 (+ 481us) client_negotiation.cc:769] Sending connection context
0928 05:27:38.664703 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 05:27:38.665232 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":234,"threads_started":1}
I20260928 05:27:38.668067 27090 heartbeater.cc:347] Connected to a master server at 127.8.124.254:44653
I20260928 05:27:38.668355 27090 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:38.669032 27090 heartbeater.cc:511] Master 127.8.124.254:44653 requested a full tablet report, sending...
I20260928 05:27:38.671029 26901 ts_manager.cc:194] Registered new tserver with Master: 602e01e037a7416ab16cae161828a6b3 (127.8.124.193:43373)
I20260928 05:27:38.672432 26901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:56263
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:27:38.706377 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:39.020334 27096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:39.020619 27096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:39.020720 27096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:39.027873 27096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:39.027984 27096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:39.028100 27096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:39.038034 27096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:39.039268 27096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:39.040499 27096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:39.052984 27102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:39.053033 27101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:39.053103 27104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:39.053581 27096 server_base.cc:1034] running on GCE node
I20260928 05:27:39.054210 27096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:39.055068 27096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:39.056280 27096 hybrid_clock.cc:648] HybridClock initialized: now 1790573259056230 us; error 38 us; skew 500 ppm
I20260928 05:27:39.056571 27096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:39 dist-test-slave-515x krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:39.059563 27096 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:27:39.060923 27096 webserver.cc:460] Webserver started at http://127.8.124.194:42057/ using document root <none> and password file <none>
I20260928 05:27:39.061563 27096 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:39.061707 27096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:39.061996 27096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:39.064791 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "a303e0e0c453440ca43028c09da9ad79"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "1045a228677053b37a63d94f31f5e62f"
server_key_iv: "bdf3fe316a73ddfd1ebb76b07e2b13a7"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.065536 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "a303e0e0c453440ca43028c09da9ad79"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "1045a228677053b37a63d94f31f5e62f"
server_key_iv: "bdf3fe316a73ddfd1ebb76b07e2b13a7"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.070744 27096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:27:39.074767 27111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.076253 27096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260928 05:27:39.076399 27096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "a303e0e0c453440ca43028c09da9ad79"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "1045a228677053b37a63d94f31f5e62f"
server_key_iv: "bdf3fe316a73ddfd1ebb76b07e2b13a7"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.076598 27096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:39.104028 27096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:39.108969 27096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:39.109241 27096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:39.110409 27096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:39.111770 27096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:39.111841 27096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.111923 27096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:39.111964 27096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.143841 27096 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:44529
I20260928 05:27:39.143884 27223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:44529 every 8 connection(s)
I20260928 05:27:39.145134 27096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:39.148775 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27096
I20260928 05:27:39.148954 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:39.149432 8691 external_mini_cluster.cc:1428] Setting key 3a6f88024d5a79995049f3651bdfcc05
Sep 28 05:27:39 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:39.164999 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.147649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:35805 (local address 127.8.124.254:44653)
0928 05:27:39.147815 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.147819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.148904 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 05:27:39.151040 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.151046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.151126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.151208 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:39.152708 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.153490 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.155418 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.155812 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.155926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.159361 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:39.159368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:39.159374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:39.159413 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:39.161899 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.162722 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.162728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.162730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.162914 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.163368 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.163372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.163374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.163641 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:39.163774 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.164564 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.164763 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:27:39.166322 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.148053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:44653 (local address 127.8.124.194:35805)
0928 05:27:39.148755 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.148793 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.150787 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.151450 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.151458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.151975 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:39.152539 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.152557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.153682 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.153687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.155276 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.155293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.156025 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.156032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.156285 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.156929 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:39.156952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:39.159130 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:39.162028 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:39.162037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:39.162055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:39.162570 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:39.163025 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:39.163030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:39.163033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:39.163258 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:39.163772 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:39.163780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:39.164190 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.164469 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.164988 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260928 05:27:39.167711 27224 heartbeater.cc:347] Connected to a master server at 127.8.124.254:44653
I20260928 05:27:39.168009 27224 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:39.168812 27224 heartbeater.cc:511] Master 127.8.124.254:44653 requested a full tablet report, sending...
I20260928 05:27:39.170430 26901 ts_manager.cc:194] Registered new tserver with Master: a303e0e0c453440ca43028c09da9ad79 (127.8.124.194:44529)
I20260928 05:27:39.171372 26901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:35805
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:27:39.204442 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:39.514757 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:39.515015 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:39.515118 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:39.522073 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:39.522183 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:39.522302 27229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:39.531845 27229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:44653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:39.533098 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:39.534355 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:39.546820 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:39.546895 27237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:39.546912 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:39.547857 27229 server_base.cc:1034] running on GCE node
I20260928 05:27:39.548494 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:39.549400 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:39.550625 27229 hybrid_clock.cc:648] HybridClock initialized: now 1790573259550564 us; error 43 us; skew 500 ppm
I20260928 05:27:39.550958 27229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:39 dist-test-slave-515x krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:39.553994 27229 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:27:39.555454 27229 webserver.cc:460] Webserver started at http://127.8.124.195:36121/ using document root <none> and password file <none>
I20260928 05:27:39.556097 27229 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:39.556217 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:39.556512 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:39.559458 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "efb50f23fcfe4a72bb95adaf86cfa12c"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "3c9625ace5c2b1851cd8b7370bf4edc4"
server_key_iv: "7f03ba46b493f9c04c9570dc19d15402"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.560209 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "efb50f23fcfe4a72bb95adaf86cfa12c"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "3c9625ace5c2b1851cd8b7370bf4edc4"
server_key_iv: "7f03ba46b493f9c04c9570dc19d15402"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.565240 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:27:39.568603 27244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.569976 27229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:39.570130 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "efb50f23fcfe4a72bb95adaf86cfa12c"
format_stamp: "Formatted at 2026-09-28 05:27:39 on dist-test-slave-515x"
server_key: "3c9625ace5c2b1851cd8b7370bf4edc4"
server_key_iv: "7f03ba46b493f9c04c9570dc19d15402"
server_key_version: "encryptionkey@0"
I20260928 05:27:39.570315 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:39.583125 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:39.587277 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:39.587541 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:39.588454 27229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:39.589759 27229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:39.589835 27229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.589944 27229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:39.589998 27229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:39.620148 27229 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:41045
I20260928 05:27:39.620203 27356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:41045 every 8 connection(s)
I20260928 05:27:39.621409 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:39.625974 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27229
I20260928 05:27:39.626173 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:39.626708 8691 external_mini_cluster.cc:1428] Setting key 16bc0f86cfe89baf36f29d1d21dec7ee
Sep 28 05:27:39 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:39.647145 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.623869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:32909 (local address 127.8.124.254:44653)
0928 05:27:39.624032 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.624037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.625192 (+ 1155us) server_negotiation.cc:407] Connection header received
0928 05:27:39.627490 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.627497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.627609 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.627695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:39.629303 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.630167 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.631666 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.632062 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.632185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.641312 (+ 9127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:39.641319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:39.641325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:39.641368 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:39.643906 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.644754 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.644761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.644765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.644921 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.645330 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.645335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.645338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.645706 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:39.645870 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.646583 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.646806 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:27:39.648041 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.624302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:44653 (local address 127.8.124.195:32909)
0928 05:27:39.624988 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.625026 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.627216 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.627936 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.627945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.628572 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:39.629145 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.629161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.630308 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.630311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.631534 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.631546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.632279 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.632286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.632529 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.638553 (+ 6024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:39.638579 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:39.641056 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:39.644097 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:39.644106 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:39.644124 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:39.644634 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:39.645023 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:39.645027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:39.645029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:39.645232 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:39.645849 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:39.645855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:39.646231 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.646469 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.646919 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260928 05:27:39.649379 27357 heartbeater.cc:347] Connected to a master server at 127.8.124.254:44653
I20260928 05:27:39.649662 27357 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:39.650337 27357 heartbeater.cc:511] Master 127.8.124.254:44653 requested a full tablet report, sending...
I20260928 05:27:39.651677 26901 ts_manager.cc:194] Registered new tserver with Master: efb50f23fcfe4a72bb95adaf86cfa12c (127.8.124.195:41045)
I20260928 05:27:39.652594 26901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:32909
I20260928 05:27:39.654104 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:39.670821 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.658278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.8.124.254:44653)
0928 05:27:39.658433 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.658437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.658883 (+ 446us) server_negotiation.cc:407] Connection header received
0928 05:27:39.659030 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.659035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.659115 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.659206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:39.660517 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.661229 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.662529 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.662932 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.663037 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.665577 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:39.665584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:39.665588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:39.665667 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:39.668038 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.668783 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.668789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.668793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.668918 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:39.669410 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:39.669414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:39.669416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:39.669763 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:39.669998 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.670393 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.670558 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:39.675302 27090 heartbeater.cc:503] Master 127.8.124.254:44653 was elected leader, sending a full tablet report...
I20260928 05:27:39.674623 26901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:39.681710 26901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:39.715574 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.692687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:43373 (local address 127.0.0.1:45486)
0928 05:27:39.703096 (+ 10409us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.703132 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.703310 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.707119 (+ 3809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.707127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.707163 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:39.707615 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.707627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.709643 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.709647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.712095 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.712111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.714001 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.714007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.714529 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.714624 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.714774 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.714954 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10175,"mutex_wait_us":6,"thread_start_us":10050,"threads_started":1}
I20260928 05:27:39.716521 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.693253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.8.124.193:43373)
0928 05:27:39.706018 (+ 12765us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.706027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.706062 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:27:39.706178 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.706187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.706489 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.706692 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:39.707826 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.709407 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.712275 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.713794 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.713922 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.714157 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.716034 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.716206 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12604,"thread_start_us":12352,"threads_started":1}
I20260928 05:27:39.717800 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.693241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.8.124.194:44529)
0928 05:27:39.709199 (+ 15958us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.709216 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.709242 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:27:39.709332 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.709340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.709585 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.710727 (+ 1142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:39.710964 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.712414 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.715270 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.716748 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.716895 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.717075 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.717314 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.717473 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15805,"thread_start_us":16502,"threads_started":1}
I20260928 05:27:39.720077 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.692706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:44529 (local address 127.0.0.1:51512)
0928 05:27:39.704367 (+ 11661us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.704383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.704484 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.709999 (+ 5515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.710004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.710035 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:39.710505 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.710519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.713160 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.713167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.715035 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.715048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.716950 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.716957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.717123 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.717171 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.717267 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.717410 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11539,"thread_start_us":11421,"threads_started":1}
I20260928 05:27:39.721048 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.704702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:41045 (local address 127.0.0.1:40600)
0928 05:27:39.712866 (+ 8164us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.712884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.712986 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.714241 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.714246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.714271 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:39.714691 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.714721 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.716493 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.716498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.718429 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.718443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.719687 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.719694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.719879 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.720506 (+ 627us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.720593 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.720726 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8048,"mutex_wait_us":22,"thread_start_us":7919,"threads_started":1}
I20260928 05:27:39.720476 27025 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfab4826e844038fc2fa4208503cc2 (DEFAULT_TABLE table=test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:39.721160 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.704796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.8.124.195:41045)
0928 05:27:39.713591 (+ 8795us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.713618 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.713651 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:27:39.713752 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.713759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.713986 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.714161 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:39.714911 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.716361 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.718597 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.719477 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.719598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.719843 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.720681 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.720837 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8629,"thread_start_us":8485,"threads_started":1}
I20260928 05:27:39.721967 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfab4826e844038fc2fa4208503cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:39.723338 27159 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfab4826e844038fc2fa4208503cc2 (DEFAULT_TABLE table=test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:39.724404 27159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfab4826e844038fc2fa4208503cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:39.724539 27292 tablet_service.cc:1467] Processing CreateTablet for tablet 47dfab4826e844038fc2fa4208503cc2 (DEFAULT_TABLE table=test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:39.726341 27292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dfab4826e844038fc2fa4208503cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:39.734714 27375 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Bootstrap starting.
I20260928 05:27:39.737749 27375 tablet_bootstrap.cc:654] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:39.738669 27375 log.cc:824] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:39.739904 27376 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Bootstrap starting.
I20260928 05:27:39.741546 27375 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: No bootstrap required, opened a new log
I20260928 05:27:39.741828 27375 ts_tablet_manager.cc:1397] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260928 05:27:39.744170 27376 tablet_bootstrap.cc:654] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:39.744984 27378 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Bootstrap starting.
I20260928 05:27:39.745319 27376 log.cc:824] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:39.747920 27378 tablet_bootstrap.cc:654] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:39.748349 27376 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: No bootstrap required, opened a new log
I20260928 05:27:39.748600 27376 ts_tablet_manager.cc:1397] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:27:39.748863 27378 log.cc:824] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:39.751207 27378 tablet_bootstrap.cc:492] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: No bootstrap required, opened a new log
I20260928 05:27:39.751422 27378 ts_tablet_manager.cc:1397] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:27:39.751351 27375 raft_consensus.cc:348] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.751772 27375 raft_consensus.cc:374] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:39.751871 27375 raft_consensus.cc:729] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303e0e0c453440ca43028c09da9ad79, State: Initialized, Role: FOLLOWER
I20260928 05:27:39.752462 27375 consensus_queue.cc:260] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.758289 27224 heartbeater.cc:503] Master 127.8.124.254:44653 was elected leader, sending a full tablet report...
I20260928 05:27:39.759002 27375 ts_tablet_manager.cc:1428] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260928 05:27:39.759529 27376 raft_consensus.cc:348] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.759964 27376 raft_consensus.cc:374] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:39.760067 27376 raft_consensus.cc:729] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602e01e037a7416ab16cae161828a6b3, State: Initialized, Role: FOLLOWER
I20260928 05:27:39.760363 27378 raft_consensus.cc:348] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.760758 27378 raft_consensus.cc:374] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:39.760677 27376 consensus_queue.cc:260] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.760848 27378 raft_consensus.cc:729] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50f23fcfe4a72bb95adaf86cfa12c, State: Initialized, Role: FOLLOWER
I20260928 05:27:39.761447 27378 consensus_queue.cc:260] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.763221 27357 heartbeater.cc:503] Master 127.8.124.254:44653 was elected leader, sending a full tablet report...
I20260928 05:27:39.763890 27378 ts_tablet_manager.cc:1428] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 05:27:39.765393 27376 ts_tablet_manager.cc:1428] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
W20260928 05:27:39.873023 27358 tablet.cc:2369] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:27:39.897095 27225 tablet.cc:2369] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:27:39.897719 27091 tablet.cc:2369] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:39.977295 27381 raft_consensus.cc:482] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:39.977640 27381 raft_consensus.cc:504] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.979269 27381 leader_election.cc:288] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 602e01e037a7416ab16cae161828a6b3 (127.8.124.193:43373), efb50f23fcfe4a72bb95adaf86cfa12c (127.8.124.195:41045)
I20260928 05:27:39.987960 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.979755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:57069 (local address 127.8.124.193:43373)
0928 05:27:39.979993 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.979998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.980748 (+ 750us) server_negotiation.cc:407] Connection header received
0928 05:27:39.980882 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.980888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.980967 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.981065 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:39.981835 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.982918 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.985749 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.986988 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.987118 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.987198 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.987581 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.987710 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:27:39.987960 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.980042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:42787 (local address 127.8.124.195:41045)
0928 05:27:39.980224 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:39.980230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:39.980644 (+ 414us) server_negotiation.cc:407] Connection header received
0928 05:27:39.980883 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:39.980888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:39.980967 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:39.981065 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:39.981802 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.982971 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.985762 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.986946 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.987116 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.987194 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:39.987581 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 05:27:39.987713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:27:39.988219 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.979659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:43373 (local address 127.8.124.194:57069)
0928 05:27:39.980662 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.980677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.980782 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.981089 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.981094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.981138 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:39.981686 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.981696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.983046 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.983051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.985581 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.985591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.987202 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.987207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.987412 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.987456 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.987553 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.987724 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":151,"spinlock_wait_cycles":7296,"thread_start_us":465,"threads_started":1}
I20260928 05:27:39.988242 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:39.979597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:41045 (local address 127.8.124.194:42787)
0928 05:27:39.980528 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:39.980548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:39.980717 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:39.981089 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:39.981094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:39.981130 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:39.981647 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.981662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.983079 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.983083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.985575 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:39.985591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:39.987195 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:39.987201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:39.987404 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:39.987465 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 05:27:39.987567 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 05:27:39.987724 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":166,"spinlock_wait_cycles":1408,"thread_start_us":791,"threads_started":1}
I20260928 05:27:39.989526 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfab4826e844038fc2fa4208503cc2" candidate_uuid: "a303e0e0c453440ca43028c09da9ad79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" is_pre_election: true
I20260928 05:27:39.989526 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfab4826e844038fc2fa4208503cc2" candidate_uuid: "a303e0e0c453440ca43028c09da9ad79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602e01e037a7416ab16cae161828a6b3" is_pre_election: true
I20260928 05:27:39.990149 27312 raft_consensus.cc:2454] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303e0e0c453440ca43028c09da9ad79 in term 0.
I20260928 05:27:39.990193 27045 raft_consensus.cc:2454] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a303e0e0c453440ca43028c09da9ad79 in term 0.
I20260928 05:27:39.990870 27114 leader_election.cc:302] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a303e0e0c453440ca43028c09da9ad79, efb50f23fcfe4a72bb95adaf86cfa12c; no voters:
I20260928 05:27:39.991345 27381 raft_consensus.cc:2781] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:39.991501 27381 raft_consensus.cc:482] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:39.991600 27381 raft_consensus.cc:3037] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:39.993914 27381 raft_consensus.cc:504] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:39.994565 27381 leader_election.cc:288] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [CANDIDATE]: Term 1 election: Requested vote from peers 602e01e037a7416ab16cae161828a6b3 (127.8.124.193:43373), efb50f23fcfe4a72bb95adaf86cfa12c (127.8.124.195:41045)
I20260928 05:27:39.995018 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfab4826e844038fc2fa4208503cc2" candidate_uuid: "a303e0e0c453440ca43028c09da9ad79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602e01e037a7416ab16cae161828a6b3"
I20260928 05:27:39.995041 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dfab4826e844038fc2fa4208503cc2" candidate_uuid: "a303e0e0c453440ca43028c09da9ad79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c"
I20260928 05:27:39.995213 27045 raft_consensus.cc:3037] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:39.995245 27312 raft_consensus.cc:3037] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:39.997323 27312 raft_consensus.cc:2454] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303e0e0c453440ca43028c09da9ad79 in term 1.
I20260928 05:27:39.997330 27045 raft_consensus.cc:2454] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a303e0e0c453440ca43028c09da9ad79 in term 1.
I20260928 05:27:39.997749 27113 leader_election.cc:302] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 602e01e037a7416ab16cae161828a6b3, a303e0e0c453440ca43028c09da9ad79; no voters:
I20260928 05:27:39.998097 27381 raft_consensus.cc:2781] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:39.998577 27381 raft_consensus.cc:686] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 1 LEADER]: Becoming Leader. State: Replica: a303e0e0c453440ca43028c09da9ad79, State: Running, Role: LEADER
I20260928 05:27:39.999047 27381 consensus_queue.cc:237] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } }
I20260928 05:27:40.003651 26901 catalog_manager.cc:5495] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 reported cstate change: term changed from 0 to 1, leader changed from <none> to a303e0e0c453440ca43028c09da9ad79 (127.8.124.194). New cstate: current_term: 1 leader_uuid: "a303e0e0c453440ca43028c09da9ad79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a303e0e0c453440ca43028c09da9ad79" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 44529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20260928 05:27:40.083266 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:40.077559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.8.124.194:44529)
0928 05:27:40.077783 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:40.077789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:40.077807 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:27:40.077891 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:40.077895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:40.077970 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:40.078097 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:40.078499 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.079210 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.080564 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.081247 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.081370 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:40.081936 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:40.082594 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:40.082762 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:40.082919 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 05:27:40.082967 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:27:40.091790 27312 raft_consensus.cc:1261] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 1 FOLLOWER]: Refusing update from remote peer a303e0e0c453440ca43028c09da9ad79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:40.091796 27045 raft_consensus.cc:1261] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 1 FOLLOWER]: Refusing update from remote peer a303e0e0c453440ca43028c09da9ad79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:40.092679 27381 consensus_queue.cc:1035] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "602e01e037a7416ab16cae161828a6b3" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:40.092952 27386 consensus_queue.cc:1035] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50f23fcfe4a72bb95adaf86cfa12c" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:40.103876 27389 mvcc.cc:205] Tried to move back new op lower bound from 7334188073326792704 to 7334188072959684608. Current Snapshot: MvccSnapshot[applied={T|T < 7334188073326792704}]
I20260928 05:27:40.120395 26901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:40.120695 26901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:40.125448 26901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c401e88e1b4d51b54a5694f7f2491e: Sending DeleteTablet for 3 replicas of tablet 47dfab4826e844038fc2fa4208503cc2
I20260928 05:27:40.126803 27159 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfab4826e844038fc2fa4208503cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:40 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:51512
I20260928 05:27:40.126905 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfab4826e844038fc2fa4208503cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:40 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:45486
I20260928 05:27:40.127027 27292 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dfab4826e844038fc2fa4208503cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:40 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:40600
I20260928 05:27:40.133143 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26962
I20260928 05:27:40.135406 27399 tablet_replica.cc:330] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: stopping tablet replica
I20260928 05:27:40.135942 27399 raft_consensus.cc:2229] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:40.136838 27399 raft_consensus.cc:2258] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:40.139658 27400 tablet_replica.cc:330] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: stopping tablet replica
I20260928 05:27:40.140035 27400 raft_consensus.cc:2229] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:40.140555 27400 raft_consensus.cc:2258] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:40.142143 27399 ts_tablet_manager.cc:1905] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:40.142688 27401 tablet_replica.cc:330] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: stopping tablet replica
I20260928 05:27:40.143113 27401 raft_consensus.cc:2229] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:40.143630 27401 raft_consensus.cc:2258] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:40.143726 27400 ts_tablet_manager.cc:1905] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:40.145359 27401 ts_tablet_manager.cc:1905] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:40.151463 27400 ts_tablet_manager.cc:1918] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:40.151669 27400 log.cc:1196] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/47dfab4826e844038fc2fa4208503cc2
I20260928 05:27:40.151858 27401 ts_tablet_manager.cc:1918] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:40.152220 27400 ts_tablet_manager.cc:1939] T 47dfab4826e844038fc2fa4208503cc2 P 602e01e037a7416ab16cae161828a6b3: Deleting consensus metadata
I20260928 05:27:40.152427 27401 log.cc:1196] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/47dfab4826e844038fc2fa4208503cc2
I20260928 05:27:40.152935 27401 ts_tablet_manager.cc:1939] T 47dfab4826e844038fc2fa4208503cc2 P efb50f23fcfe4a72bb95adaf86cfa12c: Deleting consensus metadata
I20260928 05:27:40.154238 26886 catalog_manager.cc:4841] TS 602e01e037a7416ab16cae161828a6b3 (127.8.124.193:43373): tablet 47dfab4826e844038fc2fa4208503cc2 (table test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]) successfully deleted
I20260928 05:27:40.154700 26887 catalog_manager.cc:4841] TS efb50f23fcfe4a72bb95adaf86cfa12c (127.8.124.195:41045): tablet 47dfab4826e844038fc2fa4208503cc2 (table test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]) successfully deleted
Sep 28 05:27:40 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:27:40.162127 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:40.134130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.8.124.193:43373)
0928 05:27:40.134319 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:40.134325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:40.145234 (+ 10909us) server_negotiation.cc:407] Connection header received
0928 05:27:40.145711 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:40.145718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:40.145815 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:40.145897 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:40.147622 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.148733 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.150573 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.151227 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.151316 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:40.156649 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:40.156661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:40.156675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:40.156715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:40.159370 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.160050 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.160058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.160060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.160193 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.160574 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.160577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.160579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.160926 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:40.161213 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:40.161528 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 05:27:40.161778 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:27:40.162642 27399 ts_tablet_manager.cc:1918] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:40.162783 27085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:40.162827 27399 log.cc:1196] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/47dfab4826e844038fc2fa4208503cc2
I20260928 05:27:40.163375 27399 ts_tablet_manager.cc:1939] T 47dfab4826e844038fc2fa4208503cc2 P a303e0e0c453440ca43028c09da9ad79: Deleting consensus metadata
I20260928 05:27:40.165124 26887 catalog_manager.cc:4841] TS a303e0e0c453440ca43028c09da9ad79 (127.8.124.194:44529): tablet 47dfab4826e844038fc2fa4208503cc2 (table test-table [id=7a92e6cd600c4ec99581d7c37ab4790d]) successfully deleted
I20260928 05:27:40.251951 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26962
I20260928 05:27:40.266266 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27096
Sep 28 05:27:40 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:27:40.282714 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:40.267032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.8.124.194:44529)
0928 05:27:40.267196 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:40.267208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:40.267231 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:27:40.267447 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:40.267451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:40.267517 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:40.267590 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:40.269194 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.270041 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.271820 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.272251 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.272342 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:40.277076 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:40.277090 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:40.277106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:40.277148 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:40.280011 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.280682 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.280689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.280694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.280827 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.281220 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.281223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.281226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.281572 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:40.281974 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:40.282278 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 05:27:40.282442 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:27:40.283437 27219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:40.394164 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27096
I20260928 05:27:40.409083 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27229
Sep 28 05:27:40 dist-test-slave-515x krb5kdc[26856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:27:40.424108 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:40.409866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.8.124.195:41045)
0928 05:27:40.410118 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:40.410124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:40.410266 (+ 142us) server_negotiation.cc:407] Connection header received
0928 05:27:40.410404 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:40.410409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:40.410476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:40.410562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:40.411851 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.412594 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.413903 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:40.414278 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:40.414384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:40.418642 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:40.418657 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:40.418674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:40.418717 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:40.421131 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.422076 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.422083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.422085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.422236 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:40.422586 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:40.422590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:40.422592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:40.422939 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:40.423237 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:40.423579 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 05:27:40.423789 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:27:40.424690 27352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:40.510571 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27229
I20260928 05:27:40.523662 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 26869
I20260928 05:27:40.524762 26931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:40.632800 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 26869
2026-09-28T05:27:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4187 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T05:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:40Z Disabled control of system clock
I20260928 05:27:40.700644 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46221
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:41.015688 27410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:41.015971 27410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:41.016093 27410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:41.023175 27410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:41.023274 27410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:41.023341 27410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:41.023422 27410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:41.023480 27410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:41.032481 27410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:41.033700 27410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:41.035395 27410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:41.046146 27415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.046226 27418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.046181 27416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:41.047178 27410 server_base.cc:1034] running on GCE node
I20260928 05:27:41.047910 27410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:41.049239 27410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:41.050447 27410 hybrid_clock.cc:648] HybridClock initialized: now 1790573261050399 us; error 40 us; skew 500 ppm
I20260928 05:27:41.050778 27410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:41.052860 27410 webserver.cc:460] Webserver started at http://127.8.124.254:40005/ using document root <none> and password file <none>
I20260928 05:27:41.053484 27410 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:41.053578 27410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:41.053880 27410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:41.056566 27410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "20f70cd4028f4b6db710557077e95564"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "760aaa6a56be16bbf08c3c8cc8d94c20"
server_key_iv: "672f0235ee000c78588826d8fba7f8d3"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.057298 27410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "20f70cd4028f4b6db710557077e95564"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "760aaa6a56be16bbf08c3c8cc8d94c20"
server_key_iv: "672f0235ee000c78588826d8fba7f8d3"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.062105 27410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:27:41.065376 27424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:41.066664 27410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:41.066824 27410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "20f70cd4028f4b6db710557077e95564"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "760aaa6a56be16bbf08c3c8cc8d94c20"
server_key_iv: "672f0235ee000c78588826d8fba7f8d3"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.067021 27410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:41.088654 27410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:41.089433 27410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:41.089722 27410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:41.106221 27410 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:46221
I20260928 05:27:41.106228 27475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:46221 every 8 connection(s)
I20260928 05:27:41.107470 27410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:41.110591 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27410
I20260928 05:27:41.110769 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:41.111337 8691 external_mini_cluster.cc:1428] Setting key 5c2080407c943c91daa616a6e2f3660a
I20260928 05:27:41.111371 27476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:41.123943 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:41.113040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.8.124.254:46221)
0928 05:27:41.113933 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:41.113943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:41.113985 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:27:41.115503 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:41.115524 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:41.116003 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:41.116368 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:41.116903 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.118334 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.119601 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.120256 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.120421 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:41.121660 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:41.121668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:41.121683 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:41.122255 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:41.122337 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:41.122357 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:41.122457 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:41.122673 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 05:27:41.122992 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260928 05:27:41.124049 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: Bootstrap starting.
I20260928 05:27:41.127715 27476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:41.128849 27476 log.cc:824] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:41.131489 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: No bootstrap required, opened a new log
I20260928 05:27:41.138875 27476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } }
I20260928 05:27:41.139201 27476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:41.139269 27476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f70cd4028f4b6db710557077e95564, State: Initialized, Role: FOLLOWER
I20260928 05:27:41.139775 27476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } }
I20260928 05:27:41.139971 27476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:41.140061 27476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:41.140183 27476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:41.142241 27476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } }
I20260928 05:27:41.142666 27476 leader_election.cc:302] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f70cd4028f4b6db710557077e95564; no voters:
I20260928 05:27:41.143301 27476 leader_election.cc:288] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:41.143440 27481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:41.144114 27481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [term 1 LEADER]: Becoming Leader. State: Replica: 20f70cd4028f4b6db710557077e95564, State: Running, Role: LEADER
I20260928 05:27:41.144697 27481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } }
I20260928 05:27:41.145452 27476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:41.147495 27483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f70cd4028f4b6db710557077e95564. Latest consensus state: current_term: 1 leader_uuid: "20f70cd4028f4b6db710557077e95564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } } }
I20260928 05:27:41.147467 27482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f70cd4028f4b6db710557077e95564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f70cd4028f4b6db710557077e95564" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46221 } } }
I20260928 05:27:41.147886 27483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:41.147891 27482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:41.150978 27488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:41.155985 27488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:41.164562 27488 catalog_manager.cc:1347] Generated new cluster ID: 6e0710f3695f4480812eaa009927eea3
I20260928 05:27:41.164659 27488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:41.188153 27488 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:41.189735 27488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:41.201140 27488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: Generated new TSK 0
I20260928 05:27:41.202332 27488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:41.206120 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--builtin_ntp_servers=127.8.124.212:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:41.521533 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:41.521893 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:41.522028 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:41.529326 27500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:41.529435 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:41.529574 27500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:41.538707 27500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:41.539937 27500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:41.541711 27500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:41.554409 27505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.554483 27506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.554457 27508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:41.555269 27500 server_base.cc:1034] running on GCE node
I20260928 05:27:41.555910 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:41.556887 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:41.558082 27500 hybrid_clock.cc:648] HybridClock initialized: now 1790573261558032 us; error 39 us; skew 500 ppm
I20260928 05:27:41.558385 27500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:41.560595 27500 webserver.cc:460] Webserver started at http://127.8.124.193:35163/ using document root <none> and password file <none>
I20260928 05:27:41.561277 27500 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:41.561383 27500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:41.561745 27500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:41.564554 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "2bf715be23864078871418d1022aa916"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "75828b8d489e48522bec5c034fdf6a9f"
server_key_iv: "676c4c6f4c4c19f02727290fb971f435"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.565413 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf715be23864078871418d1022aa916"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "75828b8d489e48522bec5c034fdf6a9f"
server_key_iv: "676c4c6f4c4c19f02727290fb971f435"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.570438 27500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:27:41.573709 27514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:41.575011 27500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:27:41.575160 27500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "2bf715be23864078871418d1022aa916"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "75828b8d489e48522bec5c034fdf6a9f"
server_key_iv: "676c4c6f4c4c19f02727290fb971f435"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.575353 27500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:41.586472 27500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:41.587242 27500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:41.587502 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:41.588388 27500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:41.589761 27500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:41.589861 27500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:41.589965 27500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:41.590035 27500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:41.619091 27500 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:34395
I20260928 05:27:41.619140 27626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:34395 every 8 connection(s)
I20260928 05:27:41.620371 27500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:41.627719 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27500
I20260928 05:27:41.627863 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:41.628300 8691 external_mini_cluster.cc:1428] Setting key 5fa8a1a762b4627801c6762965f540b5
I20260928 05:27:41.631070 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--builtin_ntp_servers=127.8.124.212:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:41.632771 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:41.622926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:58771 (local address 127.8.124.254:46221)
0928 05:27:41.623154 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:41.623160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:41.624430 (+ 1270us) server_negotiation.cc:407] Connection header received
0928 05:27:41.626174 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:41.626179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:41.626247 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:41.626341 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:41.627344 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.628114 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.630138 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.630571 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.630686 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:41.631990 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:41.631994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:41.631997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:41.632014 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:41.632045 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:41.632056 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:41.632131 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:41.632359 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 05:27:41.632459 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 05:27:41.633841 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:41.623416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46221 (local address 127.8.124.193:58771)
0928 05:27:41.624271 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:41.624313 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:41.625935 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:41.626532 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:41.626539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:41.626631 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:41.627198 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:41.627215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.628263 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.628267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:41.629931 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:41.629947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:41.630788 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:41.630795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:41.631009 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:41.631671 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:41.631674 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:41.631722 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:41.631802 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:41.632142 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:41.632153 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:41.632162 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 05:27:41.632351 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 05:27:41.632746 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":260,"threads_started":1}
I20260928 05:27:41.635378 27627 heartbeater.cc:347] Connected to a master server at 127.8.124.254:46221
I20260928 05:27:41.635655 27627 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:41.636380 27627 heartbeater.cc:511] Master 127.8.124.254:46221 requested a full tablet report, sending...
I20260928 05:27:41.638666 27441 ts_manager.cc:194] Registered new tserver with Master: 2bf715be23864078871418d1022aa916 (127.8.124.193:34395)
I20260928 05:27:41.640170 27441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.193:58771
W20260928 05:27:41.950608 27630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:41.950846 27630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:41.950963 27630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:41.957989 27630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:41.958072 27630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:41.958206 27630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:41.966897 27630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:41.968039 27630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:41.969726 27630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:41.981772 27635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.981822 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:41.981843 27636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:41.982547 27630 server_base.cc:1034] running on GCE node
I20260928 05:27:41.983140 27630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:41.984064 27630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:41.985267 27630 hybrid_clock.cc:648] HybridClock initialized: now 1790573261985215 us; error 47 us; skew 500 ppm
I20260928 05:27:41.985569 27630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:41.987751 27630 webserver.cc:460] Webserver started at http://127.8.124.194:41583/ using document root <none> and password file <none>
I20260928 05:27:41.988415 27630 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:41.988502 27630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:41.988801 27630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:41.991571 27630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "c99e444d60f04b358fb045b8b9b3e9ed"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "99bfe96a92776d09c7bbf6c7054db24e"
server_key_iv: "bfd1e4c83e58755c5724aa46ff7c4238"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.992331 27630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "c99e444d60f04b358fb045b8b9b3e9ed"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "99bfe96a92776d09c7bbf6c7054db24e"
server_key_iv: "bfd1e4c83e58755c5724aa46ff7c4238"
server_key_version: "encryptionkey@0"
I20260928 05:27:41.997156 27630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:27:42.000597 27644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.001966 27630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:42.002108 27630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "c99e444d60f04b358fb045b8b9b3e9ed"
format_stamp: "Formatted at 2026-09-28 05:27:41 on dist-test-slave-515x"
server_key: "99bfe96a92776d09c7bbf6c7054db24e"
server_key_iv: "bfd1e4c83e58755c5724aa46ff7c4238"
server_key_version: "encryptionkey@0"
I20260928 05:27:42.002300 27630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:42.018924 27630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:42.019726 27630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:42.020004 27630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:42.020931 27630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:42.022344 27630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:42.022439 27630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.022552 27630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:42.022619 27630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.050552 27630 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:45277
I20260928 05:27:42.050606 27756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:45277 every 8 connection(s)
I20260928 05:27:42.051838 27630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:42.055318 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27630
I20260928 05:27:42.055485 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:42.056035 8691 external_mini_cluster.cc:1428] Setting key b395c340b85d4723ed91dced2f679864
I20260928 05:27:42.058991 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--builtin_ntp_servers=127.8.124.212:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:42.064898 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.054208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:47857 (local address 127.8.124.254:46221)
0928 05:27:42.054415 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.054421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.056029 (+ 1608us) server_negotiation.cc:407] Connection header received
0928 05:27:42.058062 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.058066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.058130 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.058213 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:42.059573 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.060282 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.062270 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.062684 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.062798 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.064242 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:42.064246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:42.064250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:42.064268 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:42.064300 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:42.064312 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:42.064373 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.064576 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.064655 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:27:42.066226 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.054776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46221 (local address 127.8.124.194:47857)
0928 05:27:42.055844 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.055898 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.057813 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.058479 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.058491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.058616 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:42.059412 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.059435 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.060412 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.060417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.062142 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.062155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.062901 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.062906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.063169 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.063889 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:42.063892 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:42.063941 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:42.064045 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:42.064390 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:42.064399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:42.064406 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.064573 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.064947 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":290,"threads_started":1}
I20260928 05:27:42.067667 27757 heartbeater.cc:347] Connected to a master server at 127.8.124.254:46221
I20260928 05:27:42.067945 27757 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:42.068672 27757 heartbeater.cc:511] Master 127.8.124.254:46221 requested a full tablet report, sending...
I20260928 05:27:42.070155 27441 ts_manager.cc:194] Registered new tserver with Master: c99e444d60f04b358fb045b8b9b3e9ed (127.8.124.194:45277)
I20260928 05:27:42.071180 27441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.194:47857
W20260928 05:27:42.375928 27760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:42.376241 27760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:42.376374 27760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:42.383797 27760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:42.383914 27760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:42.384064 27760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:42.393136 27760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:42.394430 27760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:42.396181 27760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:42.408681 27768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:42.408646 27765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:42.408713 27766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:42.409366 27760 server_base.cc:1034] running on GCE node
I20260928 05:27:42.409938 27760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:42.410795 27760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:42.411985 27760 hybrid_clock.cc:648] HybridClock initialized: now 1790573262411949 us; error 30 us; skew 500 ppm
I20260928 05:27:42.412261 27760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:42.414436 27760 webserver.cc:460] Webserver started at http://127.8.124.195:46371/ using document root <none> and password file <none>
I20260928 05:27:42.415048 27760 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:42.415132 27760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:42.415400 27760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:42.418115 27760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "155fee296fd14b9db16ce34a8c982b8f"
format_stamp: "Formatted at 2026-09-28 05:27:42 on dist-test-slave-515x"
server_key: "492737b05683c623ce33fc2b7ccb2464"
server_key_iv: "827cae655949cc64467cc121f795773b"
server_key_version: "encryptionkey@0"
I20260928 05:27:42.418864 27760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "155fee296fd14b9db16ce34a8c982b8f"
format_stamp: "Formatted at 2026-09-28 05:27:42 on dist-test-slave-515x"
server_key: "492737b05683c623ce33fc2b7ccb2464"
server_key_iv: "827cae655949cc64467cc121f795773b"
server_key_version: "encryptionkey@0"
I20260928 05:27:42.423580 27760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:27:42.426832 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.428094 27760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:42.428257 27760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "155fee296fd14b9db16ce34a8c982b8f"
format_stamp: "Formatted at 2026-09-28 05:27:42 on dist-test-slave-515x"
server_key: "492737b05683c623ce33fc2b7ccb2464"
server_key_iv: "827cae655949cc64467cc121f795773b"
server_key_version: "encryptionkey@0"
I20260928 05:27:42.428454 27760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:42.455626 27760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:42.456440 27760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:42.456700 27760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:42.457679 27760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:42.459085 27760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:42.459172 27760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.459290 27760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:42.459355 27760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:42.488001 27760 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:42811
I20260928 05:27:42.488067 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:42811 every 8 connection(s)
I20260928 05:27:42.489266 27760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:42.493947 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27760
I20260928 05:27:42.494078 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:42.494480 8691 external_mini_cluster.cc:1428] Setting key 630d1d9a7ca9ec09e419d60156e10e4e
I20260928 05:27:42.501197 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.491645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:43201 (local address 127.8.124.254:46221)
0928 05:27:42.491819 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.491824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.493125 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 05:27:42.495033 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.495039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.495125 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.495245 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:42.496466 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.497273 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.498761 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.499149 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.499254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.500475 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:42.500480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:42.500483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:42.500499 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:42.500527 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:42.500538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:42.500607 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.500802 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.500892 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:27:42.502228 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.492136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46221 (local address 127.8.124.195:43201)
0928 05:27:42.492981 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.493021 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.494753 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.495482 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.495494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.495611 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:42.496286 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.496317 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.497406 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.497410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.498655 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.498667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.499350 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.499357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.499557 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.500199 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:42.500201 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:42.500241 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:42.500314 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:42.500616 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:42.500626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:42.500633 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.500780 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.501143 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260928 05:27:42.503341 27887 heartbeater.cc:347] Connected to a master server at 127.8.124.254:46221
I20260928 05:27:42.503572 27887 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:42.504213 27887 heartbeater.cc:511] Master 127.8.124.254:46221 requested a full tablet report, sending...
I20260928 05:27:42.505378 27441 ts_manager.cc:194] Registered new tserver with Master: 155fee296fd14b9db16ce34a8c982b8f (127.8.124.195:42811)
I20260928 05:27:42.506166 27441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.195:43201
I20260928 05:27:42.509112 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:42.518042 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.512797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.8.124.254:46221)
0928 05:27:42.512957 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.512963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.513415 (+ 452us) server_negotiation.cc:407] Connection header received
0928 05:27:42.513547 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.513550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.513629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.513714 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:42.514110 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.514807 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.516063 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.516437 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.516521 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.517370 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:42.517374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:42.517378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:42.517394 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:42.517424 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:42.517435 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:42.517538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.517684 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.517738 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:27:42.521361 27441 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:42.527874 27441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:42.558575 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.538766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:56054)
0928 05:27:42.549803 (+ 11037us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.549864 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.550060 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.551317 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.551326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.551358 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:42.551796 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.551808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.553434 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.553439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.555859 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.555873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.557665 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.557673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.557858 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.557912 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.558036 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.558210 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10714,"mutex_wait_us":76,"thread_start_us":10543,"threads_started":1}
I20260928 05:27:42.558845 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.539082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:42811 (local address 127.0.0.1:35620)
0928 05:27:42.549803 (+ 10721us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.549865 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.550047 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.551743 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.551751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.551777 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:42.552140 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.552151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.554128 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.554133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.556281 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.556291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.558006 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.558013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.558189 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.558221 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.558315 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.558433 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10447,"mutex_wait_us":103,"thread_start_us":10111,"threads_started":1}
I20260928 05:27:42.559175 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.539399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.8.124.195:42811)
0928 05:27:42.548897 (+ 9498us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.548908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.551313 (+ 2405us) server_negotiation.cc:407] Connection header received
0928 05:27:42.551414 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.551419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.551574 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.551733 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:42.552356 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.553985 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.556507 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.557820 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.557950 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.558205 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.558375 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.558533 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9305,"thread_start_us":9144,"threads_started":1}
I20260928 05:27:42.559487 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.539036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.8.124.193:34395)
0928 05:27:42.549153 (+ 10117us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.549162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.549976 (+ 814us) server_negotiation.cc:407] Connection header received
0928 05:27:42.550377 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.550387 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.550672 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.550951 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:42.552001 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.553258 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.556040 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.557451 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.557585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.557840 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.558993 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.559176 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9960,"thread_start_us":9843,"threads_started":1}
I20260928 05:27:42.559774 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.550310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:45277 (local address 127.0.0.1:47790)
0928 05:27:42.551049 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.551065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.551169 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.553067 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.553072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.553095 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:42.553425 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.553437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.555010 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.555016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.557067 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.557082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.559069 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.559074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.559245 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.559280 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.559368 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.559478 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":62,"thread_start_us":753,"threads_started":1}
I20260928 05:27:42.560029 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.550389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.8.124.194:45277)
0928 05:27:42.552327 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.552334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.552359 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:27:42.552445 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.552452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.552654 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.552837 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:42.553629 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.554844 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.557281 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.558596 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.558708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.558964 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.559413 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.559564 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775,"thread_start_us":1659,"threads_started":1}
I20260928 05:27:42.563265 27692 tablet_service.cc:1467] Processing CreateTablet for tablet c103538910674ae39a99df4d7ac15048 (DEFAULT_TABLE table=test-table [id=a57749d94b204248ac549bd17f29bf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:42.563273 27562 tablet_service.cc:1467] Processing CreateTablet for tablet c103538910674ae39a99df4d7ac15048 (DEFAULT_TABLE table=test-table [id=a57749d94b204248ac549bd17f29bf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:42.563292 27822 tablet_service.cc:1467] Processing CreateTablet for tablet c103538910674ae39a99df4d7ac15048 (DEFAULT_TABLE table=test-table [id=a57749d94b204248ac549bd17f29bf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:42.564486 27692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c103538910674ae39a99df4d7ac15048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:42.564551 27562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c103538910674ae39a99df4d7ac15048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:42.564544 27822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c103538910674ae39a99df4d7ac15048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:42.575423 27905 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Bootstrap starting.
I20260928 05:27:42.578737 27906 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Bootstrap starting.
I20260928 05:27:42.579016 27905 tablet_bootstrap.cc:654] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:42.580787 27905 log.cc:824] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:42.582499 27906 tablet_bootstrap.cc:654] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:42.583813 27906 log.cc:824] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:42.583992 27905 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: No bootstrap required, opened a new log
I20260928 05:27:42.584096 27907 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Bootstrap starting.
I20260928 05:27:42.584256 27905 ts_tablet_manager.cc:1397] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:27:42.586686 27906 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: No bootstrap required, opened a new log
I20260928 05:27:42.586825 27907 tablet_bootstrap.cc:654] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:42.586934 27906 ts_tablet_manager.cc:1397] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:27:42.587760 27907 log.cc:824] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:42.590270 27907 tablet_bootstrap.cc:492] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: No bootstrap required, opened a new log
I20260928 05:27:42.590490 27907 ts_tablet_manager.cc:1397] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260928 05:27:42.594614 27905 raft_consensus.cc:348] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.595052 27905 raft_consensus.cc:374] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:42.595185 27905 raft_consensus.cc:729] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99e444d60f04b358fb045b8b9b3e9ed, State: Initialized, Role: FOLLOWER
I20260928 05:27:42.595819 27905 consensus_queue.cc:260] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.596909 27757 heartbeater.cc:503] Master 127.8.124.254:46221 was elected leader, sending a full tablet report...
I20260928 05:27:42.596797 27906 raft_consensus.cc:348] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.597215 27906 raft_consensus.cc:374] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:42.597304 27906 raft_consensus.cc:729] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf715be23864078871418d1022aa916, State: Initialized, Role: FOLLOWER
I20260928 05:27:42.597571 27905 ts_tablet_manager.cc:1428] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260928 05:27:42.597617 27907 raft_consensus.cc:348] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.598088 27907 raft_consensus.cc:374] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:42.598189 27907 raft_consensus.cc:729] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155fee296fd14b9db16ce34a8c982b8f, State: Initialized, Role: FOLLOWER
I20260928 05:27:42.598321 27906 consensus_queue.cc:260] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.598749 27907 consensus_queue.cc:260] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.602699 27627 heartbeater.cc:503] Master 127.8.124.254:46221 was elected leader, sending a full tablet report...
I20260928 05:27:42.603050 27887 heartbeater.cc:503] Master 127.8.124.254:46221 was elected leader, sending a full tablet report...
I20260928 05:27:42.603382 27906 ts_tablet_manager.cc:1428] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260928 05:27:42.605141 27907 ts_tablet_manager.cc:1428] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
W20260928 05:27:42.622469 27628 tablet.cc:2369] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:42.702333 27913 raft_consensus.cc:482] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:42.702677 27913 raft_consensus.cc:504] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.706152 27913 leader_election.cc:288] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf715be23864078871418d1022aa916 (127.8.124.193:34395), c99e444d60f04b358fb045b8b9b3e9ed (127.8.124.194:45277)
I20260928 05:27:42.711309 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.705185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:54029 (local address 127.8.124.193:34395)
0928 05:27:42.705376 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.705382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.705406 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:27:42.705479 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.705485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.705590 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.705692 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:42.706396 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.707368 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.709332 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.710506 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.710623 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.710672 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.710970 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.711052 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:27:42.711436 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.705014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.8.124.195:54029)
0928 05:27:42.705212 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.705231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.705348 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.705924 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.705928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.705963 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:42.706188 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.706199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.707483 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.707488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.709177 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.709189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.710646 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.710652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.710826 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.710862 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.710955 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.711142 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":45}
I20260928 05:27:42.711982 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.705450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:54973 (local address 127.8.124.194:45277)
0928 05:27:42.705690 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.705696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.706333 (+ 637us) server_negotiation.cc:407] Connection header received
0928 05:27:42.706564 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.706568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.706634 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.706712 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:42.707231 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.708171 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.710102 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.711092 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.711236 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.711302 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.711715 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.711814 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:27:42.712065 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.705101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:45277 (local address 127.8.124.195:54973)
0928 05:27:42.706246 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.706261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:42.706425 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:42.706744 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:42.706748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:42.706773 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:42.707089 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.707101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.708324 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.708328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.709937 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:42.709952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.711233 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.711242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:42.711457 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.711533 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 05:27:42.711656 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 05:27:42.711774 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":1,"thread_start_us":755,"threads_started":1}
I20260928 05:27:42.712539 27582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c103538910674ae39a99df4d7ac15048" candidate_uuid: "155fee296fd14b9db16ce34a8c982b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf715be23864078871418d1022aa916" is_pre_election: true
I20260928 05:27:42.713060 27582 raft_consensus.cc:2454] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155fee296fd14b9db16ce34a8c982b8f in term 0.
I20260928 05:27:42.712998 27712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c103538910674ae39a99df4d7ac15048" candidate_uuid: "155fee296fd14b9db16ce34a8c982b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" is_pre_election: true
I20260928 05:27:42.713451 27712 raft_consensus.cc:2454] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155fee296fd14b9db16ce34a8c982b8f in term 0.
I20260928 05:27:42.713732 27778 leader_election.cc:302] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155fee296fd14b9db16ce34a8c982b8f, 2bf715be23864078871418d1022aa916; no voters:
I20260928 05:27:42.714107 27913 raft_consensus.cc:2781] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:42.714253 27913 raft_consensus.cc:482] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:42.714358 27913 raft_consensus.cc:3037] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:42.716387 27913 raft_consensus.cc:504] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.716912 27913 leader_election.cc:288] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [CANDIDATE]: Term 1 election: Requested vote from peers 2bf715be23864078871418d1022aa916 (127.8.124.193:34395), c99e444d60f04b358fb045b8b9b3e9ed (127.8.124.194:45277)
I20260928 05:27:42.717275 27582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c103538910674ae39a99df4d7ac15048" candidate_uuid: "155fee296fd14b9db16ce34a8c982b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf715be23864078871418d1022aa916"
I20260928 05:27:42.717392 27712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c103538910674ae39a99df4d7ac15048" candidate_uuid: "155fee296fd14b9db16ce34a8c982b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99e444d60f04b358fb045b8b9b3e9ed"
I20260928 05:27:42.717480 27582 raft_consensus.cc:3037] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:42.717577 27712 raft_consensus.cc:3037] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:42.720616 27712 raft_consensus.cc:2454] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155fee296fd14b9db16ce34a8c982b8f in term 1.
I20260928 05:27:42.720616 27582 raft_consensus.cc:2454] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155fee296fd14b9db16ce34a8c982b8f in term 1.
I20260928 05:27:42.720981 27776 leader_election.cc:302] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155fee296fd14b9db16ce34a8c982b8f, c99e444d60f04b358fb045b8b9b3e9ed; no voters:
I20260928 05:27:42.721246 27913 raft_consensus.cc:2781] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:42.721678 27913 raft_consensus.cc:686] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 1 LEADER]: Becoming Leader. State: Replica: 155fee296fd14b9db16ce34a8c982b8f, State: Running, Role: LEADER
I20260928 05:27:42.722155 27913 consensus_queue.cc:237] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } }
I20260928 05:27:42.726682 27440 catalog_manager.cc:5495] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 155fee296fd14b9db16ce34a8c982b8f (127.8.124.195). New cstate: current_term: 1 leader_uuid: "155fee296fd14b9db16ce34a8c982b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "155fee296fd14b9db16ce34a8c982b8f" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 } health_report { overall_health: UNKNOWN } } }
W20260928 05:27:42.740794 27888 tablet.cc:2369] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:27:42.754050 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.749023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.8.124.195:42811)
0928 05:27:42.749187 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.749191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.749328 (+ 137us) server_negotiation.cc:407] Connection header received
0928 05:27:42.749488 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.749491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.749573 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.749736 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:42.750043 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.750753 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.752167 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.752574 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.752661 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.753106 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:42.753587 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:42.753751 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.753838 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.753872 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:27:42.761850 27582 raft_consensus.cc:1261] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 1 FOLLOWER]: Refusing update from remote peer 155fee296fd14b9db16ce34a8c982b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:42.761850 27712 raft_consensus.cc:1261] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 1 FOLLOWER]: Refusing update from remote peer 155fee296fd14b9db16ce34a8c982b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:42.762856 27915 consensus_queue.cc:1035] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf715be23864078871418d1022aa916" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:42.763239 27913 consensus_queue.cc:1035] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99e444d60f04b358fb045b8b9b3e9ed" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:42.774246 27918 mvcc.cc:205] Tried to move back new op lower bound from 7334188084262539264 to 7334188084114345984. Current Snapshot: MvccSnapshot[applied={T|T < 7334188084262539264}]
I20260928 05:27:42.774571 27920 mvcc.cc:205] Tried to move back new op lower bound from 7334188084262539264 to 7334188084114345984. Current Snapshot: MvccSnapshot[applied={T|T < 7334188084262539264}]
I20260928 05:27:42.775748 27919 mvcc.cc:205] Tried to move back new op lower bound from 7334188084262539264 to 7334188084114345984. Current Snapshot: MvccSnapshot[applied={T|T < 7334188084262539264}]
I20260928 05:27:42.785579 27439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:42.785869 27439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:42.789429 27439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20f70cd4028f4b6db710557077e95564: Sending DeleteTablet for 3 replicas of tablet c103538910674ae39a99df4d7ac15048
I20260928 05:27:42.790640 27562 tablet_service.cc:1514] Processing DeleteTablet for tablet c103538910674ae39a99df4d7ac15048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:42 UTC) from {username='slave'} at 127.0.0.1:56054
I20260928 05:27:42.790812 27822 tablet_service.cc:1514] Processing DeleteTablet for tablet c103538910674ae39a99df4d7ac15048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:42 UTC) from {username='slave'} at 127.0.0.1:35620
I20260928 05:27:42.790863 27692 tablet_service.cc:1514] Processing DeleteTablet for tablet c103538910674ae39a99df4d7ac15048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:42 UTC) from {username='slave'} at 127.0.0.1:47790
I20260928 05:27:42.792820 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27500
I20260928 05:27:42.802119 27929 tablet_replica.cc:330] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: stopping tablet replica
I20260928 05:27:42.802619 27929 raft_consensus.cc:2229] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:42.802963 27927 tablet_replica.cc:330] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: stopping tablet replica
I20260928 05:27:42.803160 27929 raft_consensus.cc:2258] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:42.803366 27927 raft_consensus.cc:2229] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:42.804266 27928 tablet_replica.cc:330] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: stopping tablet replica
I20260928 05:27:42.804442 27927 raft_consensus.cc:2258] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:42.804662 27928 raft_consensus.cc:2229] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:42.805785 27928 raft_consensus.cc:2258] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:42.806816 27927 ts_tablet_manager.cc:1905] T c103538910674ae39a99df4d7ac15048 P 2bf715be23864078871418d1022aa916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:42.807832 27928 ts_tablet_manager.cc:1905] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:42.808548 27929 ts_tablet_manager.cc:1905] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:42.810365 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.794119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.8.124.193:34395)
0928 05:27:42.794297 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.794304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.802336 (+ 8032us) server_negotiation.cc:407] Connection header received
0928 05:27:42.802539 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.802544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.802671 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.802799 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:42.803678 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.804853 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.806650 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.807237 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.807331 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.808675 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:42.808685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:42.808713 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:42.809444 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:42.809502 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:42.809526 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:42.809633 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.809877 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.809986 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:27:42.811741 27622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:42.817847 27929 ts_tablet_manager.cc:1918] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:42.818107 27929 log.cc:1196] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/c103538910674ae39a99df4d7ac15048
I20260928 05:27:42.818725 27929 ts_tablet_manager.cc:1939] T c103538910674ae39a99df4d7ac15048 P c99e444d60f04b358fb045b8b9b3e9ed: Deleting consensus metadata
I20260928 05:27:42.819775 27928 ts_tablet_manager.cc:1918] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:42.820046 27928 log.cc:1196] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/c103538910674ae39a99df4d7ac15048
I20260928 05:27:42.820613 27928 ts_tablet_manager.cc:1939] T c103538910674ae39a99df4d7ac15048 P 155fee296fd14b9db16ce34a8c982b8f: Deleting consensus metadata
I20260928 05:27:42.820815 27426 catalog_manager.cc:4841] TS c99e444d60f04b358fb045b8b9b3e9ed (127.8.124.194:45277): tablet c103538910674ae39a99df4d7ac15048 (table test-table [id=a57749d94b204248ac549bd17f29bf9c]) successfully deleted
I20260928 05:27:42.822438 27426 catalog_manager.cc:4841] TS 155fee296fd14b9db16ce34a8c982b8f (127.8.124.195:42811): tablet c103538910674ae39a99df4d7ac15048 (table test-table [id=a57749d94b204248ac549bd17f29bf9c]) successfully deleted
I20260928 05:27:42.900748 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27500
W20260928 05:27:42.921911 27428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:34395 (error 108)
I20260928 05:27:42.922063 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27630
I20260928 05:27:42.923872 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.923391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54188)
0928 05:27:42.923564 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.923687 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:27:42.924180 27428 catalog_manager.cc:4584] TS 2bf715be23864078871418d1022aa916 (127.8.124.193:34395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c103538910674ae39a99df4d7ac15048: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
I20260928 05:27:42.930239 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.923339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.8.124.194:45277)
0928 05:27:42.923507 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:42.923513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:42.923530 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:27:42.923596 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:42.923601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:42.923717 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:42.923822 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:42.924323 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.925076 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.927195 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:42.927597 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:42.927696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:42.928809 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:42.928818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:42.928832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:42.929391 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:42.929441 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:42.929461 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:42.929543 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:42.929765 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 05:27:42.929861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:27:42.930934 27752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:42.978047 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.977444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54204)
0928 05:27:42.977722 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.977842 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 05:27:42.979224 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:42.978825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54208)
0928 05:27:42.978981 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:42.979068 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:27:43.020071 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27630
I20260928 05:27:43.033691 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27760
I20260928 05:27:43.040160 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:43.039709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54224)
0928 05:27:43.039875 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:43.039964 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:27:43.041301 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:43.040899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54234)
0928 05:27:43.041067 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:43.041160 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:27:43.041813 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:43.034436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.8.124.195:42811)
0928 05:27:43.034618 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:43.034624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:43.034854 (+ 230us) server_negotiation.cc:407] Connection header received
0928 05:27:43.035030 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:43.035034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:43.035123 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:43.035235 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:43.035652 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:43.036439 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:43.038394 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:43.038822 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:43.038914 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:43.040287 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:43.040297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:43.040315 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:43.041054 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:43.041100 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:43.041120 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:43.041226 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:43.041346 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 05:27:43.041441 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:27:43.042456 27882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:43.122051 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:43.121538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54246)
0928 05:27:43.121770 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:43.121879 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 05:27:43.123149 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:43.122722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:34395 (local address 127.0.0.1:54258)
0928 05:27:43.122930 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:43.123019 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 05:27:43.162178 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27760
I20260928 05:27:43.178251 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27410
I20260928 05:27:43.179340 27471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:43.288926 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27410
2026-09-28T05:27:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2655 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:43 dist-test-slave-515x krb5kdc[27938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:43 dist-test-slave-515x krb5kdc[27938](info): set up 2 sockets
Sep 28 05:27:43 dist-test-slave-515x krb5kdc[27938](info): commencing operation
krb5kdc: starting...
W20260928 05:27:44.305574 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:44 dist-test-slave-515x krb5kdc[27938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:27:44.544193 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38789
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:38789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:44.864853 27951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:44.865120 27951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:44.865259 27951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:44.872453 27951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:44.872558 27951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:44.872633 27951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:44.872709 27951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:44.872766 27951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:44.882550 27951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:38789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:44.883796 27951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:44.885068 27951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:44.896296 27956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:44.896337 27957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:44.896358 27959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:44.897120 27951 server_base.cc:1034] running on GCE node
I20260928 05:27:44.897907 27951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:44.899456 27951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:44.900681 27951 hybrid_clock.cc:648] HybridClock initialized: now 1790573264900620 us; error 31 us; skew 500 ppm
I20260928 05:27:44.901021 27951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:44 dist-test-slave-515x krb5kdc[27938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:44.907987 27951 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:27:44.909415 27951 webserver.cc:460] Webserver started at http://127.8.124.254:46381/ using document root <none> and password file <none>
I20260928 05:27:44.910121 27951 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:44.910228 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:44.910516 27951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:44.913374 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "abf590a646414d5fb32e4c36c91c5848"
format_stamp: "Formatted at 2026-09-28 05:27:44 on dist-test-slave-515x"
server_key: "498ab52ae204ef1d7efa4daebdef033c"
server_key_iv: "6f4f354a68610f6a0333d13451aaf73e"
server_key_version: "encryptionkey@0"
I20260928 05:27:44.914122 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "abf590a646414d5fb32e4c36c91c5848"
format_stamp: "Formatted at 2026-09-28 05:27:44 on dist-test-slave-515x"
server_key: "498ab52ae204ef1d7efa4daebdef033c"
server_key_iv: "6f4f354a68610f6a0333d13451aaf73e"
server_key_version: "encryptionkey@0"
I20260928 05:27:44.919237 27951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:27:44.922483 27966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:44.923667 27951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:27:44.923825 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "abf590a646414d5fb32e4c36c91c5848"
format_stamp: "Formatted at 2026-09-28 05:27:44 on dist-test-slave-515x"
server_key: "498ab52ae204ef1d7efa4daebdef033c"
server_key_iv: "6f4f354a68610f6a0333d13451aaf73e"
server_key_version: "encryptionkey@0"
I20260928 05:27:44.923990 27951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:44.945120 27951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:44.949044 27951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:44.949295 27951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:44.966300 27951 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:38789
I20260928 05:27:44.966372 28017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:38789 every 8 connection(s)
I20260928 05:27:44.967628 27951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:44.971432 28018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:44.976789 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 27951
I20260928 05:27:44.976969 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:44.977399 8691 external_mini_cluster.cc:1428] Setting key 63a09f00c82ec53754d0678497c52916
I20260928 05:27:44.981994 28018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: Bootstrap starting.
I20260928 05:27:44.985530 28018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:44.986773 28018 log.cc:824] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:44.990082 28018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: No bootstrap required, opened a new log
Sep 28 05:27:44 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:44.999862 28018 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } }
I20260928 05:27:45.000311 28018 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:45.000416 28018 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf590a646414d5fb32e4c36c91c5848, State: Initialized, Role: FOLLOWER
I20260928 05:27:45.001065 28018 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } }
I20260928 05:27:45.001356 28018 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:45.001475 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:44.979193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.8.124.254:38789)
0928 05:27:44.979984 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:44.979996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:44.980036 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:27:44.981706 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:44.981728 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:44.982105 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:44.982442 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:44.984354 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:44.985650 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:44.987428 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:44.988374 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:44.988505 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:44.993150 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:44.993160 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:44.993179 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:44.993232 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:44.997015 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:44.997737 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:44.997746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:44.997755 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:44.997977 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:44.998299 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:44.998304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:44.998308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:44.999006 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:44.999274 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:44.999704 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 05:27:45.000130 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260928 05:27:45.001499 28018 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:45.001713 28018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:45.004034 28018 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } }
I20260928 05:27:45.004601 28018 leader_election.cc:302] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf590a646414d5fb32e4c36c91c5848; no voters:
I20260928 05:27:45.005111 28018 leader_election.cc:288] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:45.005239 28023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:45.006229 28023 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [term 1 LEADER]: Becoming Leader. State: Replica: abf590a646414d5fb32e4c36c91c5848, State: Running, Role: LEADER
I20260928 05:27:45.006867 28023 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } }
I20260928 05:27:45.006989 28018 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:45.009507 28024 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf590a646414d5fb32e4c36c91c5848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } } }
I20260928 05:27:45.009573 28025 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf590a646414d5fb32e4c36c91c5848. Latest consensus state: current_term: 1 leader_uuid: "abf590a646414d5fb32e4c36c91c5848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf590a646414d5fb32e4c36c91c5848" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 38789 } } }
I20260928 05:27:45.009931 28024 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:45.009953 28025 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:45.012005 28030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:45.017506 28030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:45.026371 28030 catalog_manager.cc:1347] Generated new cluster ID: f646c76a239e4370aa982a788c4415ef
I20260928 05:27:45.026479 28030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:45.046286 28030 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:45.048234 28030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:45.056501 28030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: Generated new TSK 0
I20260928 05:27:45.057586 28030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.124.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
Entry for principal kudu/127.8.124.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab.
I20260928 05:27:45.121068 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:45.446740 28044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:45.447029 28044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:45.447163 28044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:45.454324 28044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:45.454443 28044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:45.454592 28044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:45.463971 28044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.193.keytab
--principal=kudu/127.8.124.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:45.465209 28044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:45.466527 28044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:45.479480 28050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:45.479413 28052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:45.479907 28049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:45.480368 28044 server_base.cc:1034] running on GCE node
I20260928 05:27:45.480968 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:45.481950 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:45.483178 28044 hybrid_clock.cc:648] HybridClock initialized: now 1790573265483115 us; error 52 us; skew 500 ppm
I20260928 05:27:45.483523 28044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:45 dist-test-slave-515x krb5kdc[27938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573265, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:45.486694 28044 init.cc:377] Logged in from keytab as kudu/127.8.124.193@KRBTEST.COM (short username kudu)
I20260928 05:27:45.488073 28044 webserver.cc:460] Webserver started at http://127.8.124.193:34559/ using document root <none> and password file <none>
I20260928 05:27:45.488830 28044 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:45.488942 28044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:45.489209 28044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:45.492138 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "e22bf12ae1854d039a47d536ebea7ffa"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "095901000eb2674088b78e9d46cacbee"
server_key_iv: "842d010b962658d077188eedaa762372"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.492887 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "e22bf12ae1854d039a47d536ebea7ffa"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "095901000eb2674088b78e9d46cacbee"
server_key_iv: "842d010b962658d077188eedaa762372"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.498132 28044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:27:45.501794 28059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:45.503178 28044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:45.503350 28044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "e22bf12ae1854d039a47d536ebea7ffa"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "095901000eb2674088b78e9d46cacbee"
server_key_iv: "842d010b962658d077188eedaa762372"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.503553 28044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:45.527666 28044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:45.531759 28044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:45.532022 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:45.533046 28044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:45.534415 28044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:45.534487 28044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:45.534595 28044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:45.534647 28044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:45.563517 28044 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:36165
I20260928 05:27:45.563568 28171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:36165 every 8 connection(s)
I20260928 05:27:45.564821 28044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:45.570425 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28044
I20260928 05:27:45.570595 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:45.571117 8691 external_mini_cluster.cc:1428] Setting key 23732b2a24984d6aa29da4b76ce0e1c4
Sep 28 05:27:45 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573265, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.193@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:45.586673 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:45.567374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:51275 (local address 127.8.124.254:38789)
0928 05:27:45.567816 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:45.567820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:45.568633 (+ 813us) server_negotiation.cc:407] Connection header received
0928 05:27:45.570690 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:45.570697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:45.570773 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:45.570873 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:45.572935 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:45.573986 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:45.576268 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:45.576671 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:45.576781 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:45.580884 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:45.580890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:45.580893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:45.580930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:45.583640 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:45.584369 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:45.584373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:45.584376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:45.584500 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:45.584901 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:45.584905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:45.584907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:45.585173 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:45.585308 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:45.586073 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 05:27:45.586317 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
I20260928 05:27:45.587698 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:45.567798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38789 (local address 127.8.124.193:51275)
0928 05:27:45.568474 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:45.568512 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:45.570378 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:45.571147 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:45.571159 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:45.571836 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:45.572749 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:45.572781 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:45.574162 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:45.574167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:45.576104 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:45.576120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:45.576909 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:45.576915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:45.577183 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:45.577925 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:45.577952 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:45.580622 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:45.583790 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:45.583797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:45.583813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:45.584241 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:45.584607 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:45.584611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:45.584613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:45.584796 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:45.585311 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:45.585319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:45.585694 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 05:27:45.585952 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 05:27:45.586535 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":257,"threads_started":1}
I20260928 05:27:45.589388 28172 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38789
I20260928 05:27:45.589708 28172 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:45.590428 28172 heartbeater.cc:511] Master 127.8.124.254:38789 requested a full tablet report, sending...
I20260928 05:27:45.593202 27983 ts_manager.cc:194] Registered new tserver with Master: e22bf12ae1854d039a47d536ebea7ffa (127.8.124.193:36165)
I20260928 05:27:45.595336 27983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.193@KRBTEST.COM'} at 127.8.124.193:51275
WARNING: no policy specified for kudu/127.8.124.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
Entry for principal kudu/127.8.124.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab.
I20260928 05:27:45.629142 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:45.942323 28178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:45.942579 28178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:45.942708 28178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:45.949723 28178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:45.949821 28178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:45.949958 28178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:45.959532 28178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.194.keytab
--principal=kudu/127.8.124.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:45.960705 28178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:45.961973 28178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:45.974057 28183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:45.974087 28184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:45.975238 28186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:45.976040 28178 server_base.cc:1034] running on GCE node
I20260928 05:27:45.976667 28178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:45.977638 28178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:45.978859 28178 hybrid_clock.cc:648] HybridClock initialized: now 1790573265978799 us; error 54 us; skew 500 ppm
I20260928 05:27:45.979159 28178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:45 dist-test-slave-515x krb5kdc[27938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573265, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:45.982235 28178 init.cc:377] Logged in from keytab as kudu/127.8.124.194@KRBTEST.COM (short username kudu)
I20260928 05:27:45.983670 28178 webserver.cc:460] Webserver started at http://127.8.124.194:45905/ using document root <none> and password file <none>
I20260928 05:27:45.984325 28178 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:45.984419 28178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:45.984669 28178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:45.987557 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "162c77a9402342e2860d70f5372a1126"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "f0e9e61d5a18ccb8ab44e78dd4eb195d"
server_key_iv: "c453d4137ad2e0c4405e7836bfa43920"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.988302 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "162c77a9402342e2860d70f5372a1126"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "f0e9e61d5a18ccb8ab44e78dd4eb195d"
server_key_iv: "c453d4137ad2e0c4405e7836bfa43920"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.993451 28178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:27:45.996954 28193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:45.998291 28178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:45.998433 28178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "162c77a9402342e2860d70f5372a1126"
format_stamp: "Formatted at 2026-09-28 05:27:45 on dist-test-slave-515x"
server_key: "f0e9e61d5a18ccb8ab44e78dd4eb195d"
server_key_iv: "c453d4137ad2e0c4405e7836bfa43920"
server_key_version: "encryptionkey@0"
I20260928 05:27:45.998608 28178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:46.017493 28178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:46.021430 28178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:46.021703 28178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:46.022619 28178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:46.023887 28178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:46.023957 28178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:46.024065 28178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:46.024116 28178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:46.053205 28178 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:40199
I20260928 05:27:46.053249 28305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:40199 every 8 connection(s)
I20260928 05:27:46.054459 28178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:46.061316 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28178
I20260928 05:27:46.061478 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:46.062038 8691 external_mini_cluster.cc:1428] Setting key dac3cc377032e692816ecda7fec13377
Sep 28 05:27:46 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573265, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.194@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:46.074452 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.056784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:58385 (local address 127.8.124.254:38789)
0928 05:27:46.056964 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.056968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.058087 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 05:27:46.059892 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.059897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.059963 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.060048 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:46.061452 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.062499 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.064513 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.065112 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.065230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.068519 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:46.068525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:46.068529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:46.068566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:46.071586 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.072235 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.072239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.072243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.072386 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.072840 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.072842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.072845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.073106 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:46.073210 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.074016 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.074193 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:27:46.075764 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.057246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38789 (local address 127.8.124.194:58385)
0928 05:27:46.057936 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.057976 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.059617 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.060238 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.060246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.060734 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:46.061287 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.061303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.062665 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.062687 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.064368 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.064381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.065295 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.065300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.065494 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.066128 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:46.066155 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:46.068302 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:46.071719 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:46.071726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:46.071740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:46.072137 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:46.072488 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:46.072492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:46.072495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:46.072747 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:46.073214 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:46.073222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:46.073659 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.073950 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.074460 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":268,"threads_started":1}
I20260928 05:27:46.077194 28306 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38789
I20260928 05:27:46.077479 28306 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:46.078193 28306 heartbeater.cc:511] Master 127.8.124.254:38789 requested a full tablet report, sending...
I20260928 05:27:46.079525 27983 ts_manager.cc:194] Registered new tserver with Master: 162c77a9402342e2860d70f5372a1126 (127.8.124.194:40199)
I20260928 05:27:46.080277 27983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.194@KRBTEST.COM'} at 127.8.124.194:58385
WARNING: no policy specified for kudu/127.8.124.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
Entry for principal kudu/127.8.124.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab.
I20260928 05:27:46.114202 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:46.424876 28311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:46.425168 28311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:46.425316 28311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:46.432688 28311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:46.432822 28311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:46.432982 28311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:46.442830 28311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.195.keytab
--principal=kudu/127.8.124.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:38789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:46.444111 28311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:46.445395 28311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:46.457535 28316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:46.457614 28319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:46.457643 28317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:46.458568 28311 server_base.cc:1034] running on GCE node
I20260928 05:27:46.459123 28311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:46.460026 28311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:46.461223 28311 hybrid_clock.cc:648] HybridClock initialized: now 1790573266461170 us; error 43 us; skew 500 ppm
I20260928 05:27:46.461499 28311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:46 dist-test-slave-515x krb5kdc[27938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573266, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:46.464385 28311 init.cc:377] Logged in from keytab as kudu/127.8.124.195@KRBTEST.COM (short username kudu)
I20260928 05:27:46.465710 28311 webserver.cc:460] Webserver started at http://127.8.124.195:43875/ using document root <none> and password file <none>
I20260928 05:27:46.466352 28311 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:46.466463 28311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:46.466743 28311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:46.469463 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "94021de728a44af1a7f4158a6a75fe46"
format_stamp: "Formatted at 2026-09-28 05:27:46 on dist-test-slave-515x"
server_key: "1082adb29abd649de079e1b937cef10b"
server_key_iv: "60b774923de5ff9bc65e8997a47b0d79"
server_key_version: "encryptionkey@0"
I20260928 05:27:46.470237 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "94021de728a44af1a7f4158a6a75fe46"
format_stamp: "Formatted at 2026-09-28 05:27:46 on dist-test-slave-515x"
server_key: "1082adb29abd649de079e1b937cef10b"
server_key_iv: "60b774923de5ff9bc65e8997a47b0d79"
server_key_version: "encryptionkey@0"
I20260928 05:27:46.474848 28311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 05:27:46.478055 28326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:46.479251 28311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:27:46.479394 28311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "94021de728a44af1a7f4158a6a75fe46"
format_stamp: "Formatted at 2026-09-28 05:27:46 on dist-test-slave-515x"
server_key: "1082adb29abd649de079e1b937cef10b"
server_key_iv: "60b774923de5ff9bc65e8997a47b0d79"
server_key_version: "encryptionkey@0"
I20260928 05:27:46.479570 28311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:46.509248 28311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:46.513743 28311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:46.514024 28311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:46.515092 28311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:46.516528 28311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:46.516602 28311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:46.516686 28311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:46.516736 28311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:46.546315 28311 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:43393
I20260928 05:27:46.546366 28438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:43393 every 8 connection(s)
I20260928 05:27:46.547608 28311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:46.556257 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28311
I20260928 05:27:46.556453 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:46.557058 8691 external_mini_cluster.cc:1428] Setting key 3aa88798b0974eb7ca53cb931de4db21
Sep 28 05:27:46 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573266, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.195@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:27:46.568105 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.550046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:54387 (local address 127.8.124.254:38789)
0928 05:27:46.550296 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.550301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.551426 (+ 1125us) server_negotiation.cc:407] Connection header received
0928 05:27:46.553212 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.553216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.553284 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.553373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:46.554881 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.555649 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.557968 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.558562 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.558686 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.562439 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:46.562444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:46.562448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:46.562484 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:46.565020 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.565763 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.565767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.565770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.565901 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.566321 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.566324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.566327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.566634 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:46.566778 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.567667 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.567849 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 05:27:46.569190 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.550526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:38789 (local address 127.8.124.195:54387)
0928 05:27:46.551215 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.551289 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.552940 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.553578 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.553587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.554160 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:46.554724 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.554744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.555836 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.555842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.557815 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.557831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.558810 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.558819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.559098 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.559884 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:27:46.559915 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:46.562242 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:46.565162 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:46.565171 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:46.565188 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:46.565645 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:46.566003 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:27:46.566009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:27:46.566013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:27:46.566205 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:27:46.566790 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:46.566800 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:46.567286 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.567586 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.568118 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":275,"threads_started":1}
I20260928 05:27:46.570696 28439 heartbeater.cc:347] Connected to a master server at 127.8.124.254:38789
I20260928 05:27:46.571009 28439 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:46.571699 28439 heartbeater.cc:511] Master 127.8.124.254:38789 requested a full tablet report, sending...
I20260928 05:27:46.573114 27983 ts_manager.cc:194] Registered new tserver with Master: 94021de728a44af1a7f4158a6a75fe46 (127.8.124.195:43393)
I20260928 05:27:46.573932 27983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.124.195@KRBTEST.COM'} at 127.8.124.195:54387
I20260928 05:27:46.584455 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:46.598835 28172 heartbeater.cc:503] Master 127.8.124.254:38789 was elected leader, sending a full tablet report...
I20260928 05:27:46.600508 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.588357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.8.124.254:38789)
0928 05:27:46.588535 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.588541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.588912 (+ 371us) server_negotiation.cc:407] Connection header received
0928 05:27:46.589078 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.589084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.589155 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.589233 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:46.590611 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.591324 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.592548 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.592962 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.593073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.595359 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:46.595364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:46.595368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:46.595400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:46.597800 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.598446 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.598451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.598455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.598609 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.598954 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.598959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.598963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.599378 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:46.599633 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.599940 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.600158 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:27:46.604104 27983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:46.611151 27983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:46.632082 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.623093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36165 (local address 127.0.0.1:54514)
0928 05:27:46.623822 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.623865 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.624033 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.625009 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.625016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.625047 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:46.625546 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.625559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.627496 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.627501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.629630 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.629644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.631088 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.631093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.631285 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.631340 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.631468 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.631643 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":4,"thread_start_us":342,"threads_started":1}
I20260928 05:27:46.632700 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.623268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.8.124.193:36165)
0928 05:27:46.623935 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.623943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.623972 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:27:46.624256 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.624266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.624524 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.624700 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:46.625765 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.627340 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.629768 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.630887 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.631033 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.631252 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.631523 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.631680 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":344,"threads_started":1}
I20260928 05:27:46.632788 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.624133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:40199 (local address 127.0.0.1:48018)
0928 05:27:46.624596 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.624615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.624741 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.625756 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.625760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.625785 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:46.626218 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.626230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.628163 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.628168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.630333 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.630347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.632018 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.632024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.632179 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.632215 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.632273 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.632387 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"spinlock_wait_cycles":17664,"thread_start_us":221,"threads_started":1}
I20260928 05:27:46.633380 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.624657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:43393 (local address 127.0.0.1:59648)
0928 05:27:46.625715 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.625737 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.625825 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.626932 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.626936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.626959 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:46.627321 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.627330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.629259 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.629264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.630988 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.631002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.632693 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.632701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.632876 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.632907 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.632966 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.633075 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":721,"threads_started":1}
I20260928 05:27:46.634076 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.625114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.8.124.195:43393)
0928 05:27:46.626497 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.626503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.626526 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:27:46.626594 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.626598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.626745 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.626892 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:46.627481 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.629088 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.631169 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.632469 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.633331 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.633538 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.633642 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.633800 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":974,"threads_started":1}
I20260928 05:27:46.634513 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.624379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.8.124.194:40199)
0928 05:27:46.625173 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.625182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.625210 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:27:46.625304 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.625310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.625523 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.626381 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:46.626587 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.628031 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.630492 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.631779 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.633971 (+ 2192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.634100 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.634163 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.634311 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":539,"threads_started":1}
I20260928 05:27:46.636513 28107 tablet_service.cc:1467] Processing CreateTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb (DEFAULT_TABLE table=test-table [id=31f29d61c56347269a2b1be1042d9fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:46.637288 28374 tablet_service.cc:1467] Processing CreateTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb (DEFAULT_TABLE table=test-table [id=31f29d61c56347269a2b1be1042d9fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:46.637866 28107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595e0c778f264ccc8fe0ec5e16678aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:46.638185 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb (DEFAULT_TABLE table=test-table [id=31f29d61c56347269a2b1be1042d9fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:46.638706 28374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595e0c778f264ccc8fe0ec5e16678aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:46.639469 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595e0c778f264ccc8fe0ec5e16678aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:46.648842 28457 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Bootstrap starting.
I20260928 05:27:46.651839 28457 tablet_bootstrap.cc:654] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:46.652985 28457 log.cc:824] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:46.653898 28459 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Bootstrap starting.
I20260928 05:27:46.656356 28458 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Bootstrap starting.
I20260928 05:27:46.657454 28459 tablet_bootstrap.cc:654] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:46.657882 28457 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: No bootstrap required, opened a new log
I20260928 05:27:46.658111 28457 ts_tablet_manager.cc:1397] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 05:27:46.658658 28459 log.cc:824] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:46.659575 28458 tablet_bootstrap.cc:654] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:46.660658 28458 log.cc:824] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:46.661660 28459 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: No bootstrap required, opened a new log
I20260928 05:27:46.661890 28459 ts_tablet_manager.cc:1397] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 05:27:46.663507 28458 tablet_bootstrap.cc:492] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: No bootstrap required, opened a new log
I20260928 05:27:46.663776 28458 ts_tablet_manager.cc:1397] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:27:46.667338 28457 raft_consensus.cc:348] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.667785 28457 raft_consensus.cc:374] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:46.667898 28457 raft_consensus.cc:729] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22bf12ae1854d039a47d536ebea7ffa, State: Initialized, Role: FOLLOWER
I20260928 05:27:46.668499 28457 consensus_queue.cc:260] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.669214 28459 raft_consensus.cc:348] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.669682 28459 raft_consensus.cc:374] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:46.669821 28459 raft_consensus.cc:729] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94021de728a44af1a7f4158a6a75fe46, State: Initialized, Role: FOLLOWER
I20260928 05:27:46.670208 28457 ts_tablet_manager.cc:1428] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 05:27:46.670567 28459 consensus_queue.cc:260] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.671558 28439 heartbeater.cc:503] Master 127.8.124.254:38789 was elected leader, sending a full tablet report...
I20260928 05:27:46.672153 28459 ts_tablet_manager.cc:1428] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260928 05:27:46.673061 28458 raft_consensus.cc:348] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.673432 28458 raft_consensus.cc:374] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:46.673522 28458 raft_consensus.cc:729] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162c77a9402342e2860d70f5372a1126, State: Initialized, Role: FOLLOWER
I20260928 05:27:46.674041 28458 consensus_queue.cc:260] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.675067 28306 heartbeater.cc:503] Master 127.8.124.254:38789 was elected leader, sending a full tablet report...
I20260928 05:27:46.675992 28458 ts_tablet_manager.cc:1428] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 05:27:46.710924 28464 raft_consensus.cc:482] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:46.711216 28464 raft_consensus.cc:504] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.712863 28464 leader_election.cc:288] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e22bf12ae1854d039a47d536ebea7ffa (127.8.124.193:36165), 162c77a9402342e2860d70f5372a1126 (127.8.124.194:40199)
I20260928 05:27:46.718958 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.713147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:36165 (local address 127.8.124.195:44691)
0928 05:27:46.713334 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.713352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.713477 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.714031 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.714036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.714059 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:46.714229 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.714240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.715440 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.715446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.716913 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.716925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.718223 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.718228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.718418 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.718453 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.718550 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.718668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":19}
I20260928 05:27:46.718962 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.713188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:44691 (local address 127.8.124.193:36165)
0928 05:27:46.713368 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.713372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.713392 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:27:46.713648 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.713655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.713733 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.713847 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:46.714373 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.715322 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.717063 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.718081 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.718203 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.718255 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.718595 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.718711 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:27:46.719493 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.713769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:49509 (local address 127.8.124.194:40199)
0928 05:27:46.713971 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.713977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.714187 (+ 210us) server_negotiation.cc:407] Connection header received
0928 05:27:46.714316 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.714321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.714389 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.714456 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:46.715007 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.715861 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.717703 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.718602 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.718744 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.718810 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.719189 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.719310 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 05:27:46.720155 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.713520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:40199 (local address 127.8.124.195:49509)
0928 05:27:46.714099 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:46.714112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:46.714213 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:46.714489 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:46.714493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:46.714516 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:46.714869 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.714881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.715997 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.716000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.717554 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:46.717568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.718805 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.718811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:46.719008 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.719039 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:27:46.719664 (+ 625us) client_negotiation.cc:240] Negotiation successful
0928 05:27:46.719811 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1,"thread_start_us":339,"threads_started":1}
I20260928 05:27:46.719954 28127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595e0c778f264ccc8fe0ec5e16678aeb" candidate_uuid: "94021de728a44af1a7f4158a6a75fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22bf12ae1854d039a47d536ebea7ffa" is_pre_election: true
I20260928 05:27:46.720534 28127 raft_consensus.cc:2454] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94021de728a44af1a7f4158a6a75fe46 in term 0.
I20260928 05:27:46.721019 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595e0c778f264ccc8fe0ec5e16678aeb" candidate_uuid: "94021de728a44af1a7f4158a6a75fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162c77a9402342e2860d70f5372a1126" is_pre_election: true
I20260928 05:27:46.721102 28327 leader_election.cc:302] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94021de728a44af1a7f4158a6a75fe46, e22bf12ae1854d039a47d536ebea7ffa; no voters:
I20260928 05:27:46.721423 28464 raft_consensus.cc:2781] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:46.721470 28261 raft_consensus.cc:2454] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94021de728a44af1a7f4158a6a75fe46 in term 0.
I20260928 05:27:46.721567 28464 raft_consensus.cc:482] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:46.721674 28464 raft_consensus.cc:3037] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:46.724093 28464 raft_consensus.cc:504] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.724651 28464 leader_election.cc:288] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [CANDIDATE]: Term 1 election: Requested vote from peers e22bf12ae1854d039a47d536ebea7ffa (127.8.124.193:36165), 162c77a9402342e2860d70f5372a1126 (127.8.124.194:40199)
I20260928 05:27:46.725168 28127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595e0c778f264ccc8fe0ec5e16678aeb" candidate_uuid: "94021de728a44af1a7f4158a6a75fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22bf12ae1854d039a47d536ebea7ffa"
I20260928 05:27:46.725256 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595e0c778f264ccc8fe0ec5e16678aeb" candidate_uuid: "94021de728a44af1a7f4158a6a75fe46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162c77a9402342e2860d70f5372a1126"
I20260928 05:27:46.725368 28127 raft_consensus.cc:3037] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:46.725458 28261 raft_consensus.cc:3037] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:46.727670 28261 raft_consensus.cc:2454] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94021de728a44af1a7f4158a6a75fe46 in term 1.
I20260928 05:27:46.727676 28127 raft_consensus.cc:2454] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94021de728a44af1a7f4158a6a75fe46 in term 1.
I20260928 05:27:46.728036 28327 leader_election.cc:302] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94021de728a44af1a7f4158a6a75fe46, e22bf12ae1854d039a47d536ebea7ffa; no voters:
I20260928 05:27:46.728314 28464 raft_consensus.cc:2781] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:46.728717 28464 raft_consensus.cc:686] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 1 LEADER]: Becoming Leader. State: Replica: 94021de728a44af1a7f4158a6a75fe46, State: Running, Role: LEADER
I20260928 05:27:46.729159 28464 consensus_queue.cc:237] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } }
I20260928 05:27:46.733927 27983 catalog_manager.cc:5495] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94021de728a44af1a7f4158a6a75fe46 (127.8.124.195). New cstate: current_term: 1 leader_uuid: "94021de728a44af1a7f4158a6a75fe46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94021de728a44af1a7f4158a6a75fe46" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 43393 } health_report { overall_health: HEALTHY } } }
I20260928 05:27:46.758114 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.752482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.8.124.195:43393)
0928 05:27:46.752707 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.752712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.752764 (+ 52us) server_negotiation.cc:407] Connection header received
0928 05:27:46.752846 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.752849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.752934 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.753061 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:46.753450 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.754188 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.755972 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.756374 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.756489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.757004 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:46.757512 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:46.757671 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.757770 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.757864 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:27:46.766027 28261 raft_consensus.cc:1261] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 1 FOLLOWER]: Refusing update from remote peer 94021de728a44af1a7f4158a6a75fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:46.766027 28127 raft_consensus.cc:1261] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 1 FOLLOWER]: Refusing update from remote peer 94021de728a44af1a7f4158a6a75fe46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:46.766979 28467 consensus_queue.cc:1035] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22bf12ae1854d039a47d536ebea7ffa" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 36165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:46.767362 28464 consensus_queue.cc:1035] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162c77a9402342e2860d70f5372a1126" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:46.779529 28470 mvcc.cc:205] Tried to move back new op lower bound from 7334188100663566336 to 7334188100526497792. Current Snapshot: MvccSnapshot[applied={T|T < 7334188100663566336}]
I20260928 05:27:46.785189 28471 mvcc.cc:205] Tried to move back new op lower bound from 7334188100663566336 to 7334188100526497792. Current Snapshot: MvccSnapshot[applied={T|T < 7334188100663566336}]
I20260928 05:27:46.792800 27982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:46.793051 27982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:46.796615 27982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abf590a646414d5fb32e4c36c91c5848: Sending DeleteTablet for 3 replicas of tablet 595e0c778f264ccc8fe0ec5e16678aeb
I20260928 05:27:46.797737 28107 tablet_service.cc:1514] Processing DeleteTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:46 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:54514
I20260928 05:27:46.797781 28241 tablet_service.cc:1514] Processing DeleteTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:46 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:48018
I20260928 05:27:46.798456 28374 tablet_service.cc:1514] Processing DeleteTablet for tablet 595e0c778f264ccc8fe0ec5e16678aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:46 UTC) from {username='kudu', principal='kudu/127.8.124.254@KRBTEST.COM'} at 127.0.0.1:59648
I20260928 05:27:46.799759 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28044
I20260928 05:27:46.809425 28480 tablet_replica.cc:330] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: stopping tablet replica
I20260928 05:27:46.809922 28480 raft_consensus.cc:2229] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:46.810490 28480 raft_consensus.cc:2258] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:46.811257 28482 tablet_replica.cc:330] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: stopping tablet replica
I20260928 05:27:46.811666 28482 raft_consensus.cc:2229] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:46.812446 28482 raft_consensus.cc:2258] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:46.814888 28482 ts_tablet_manager.cc:1905] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:46.818956 28481 tablet_replica.cc:330] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: stopping tablet replica
I20260928 05:27:46.820334 28480 ts_tablet_manager.cc:1905] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:27:46 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.193@KRBTEST.COM
I20260928 05:27:46.823340 28482 ts_tablet_manager.cc:1918] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:46.823539 28482 log.cc:1196] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/595e0c778f264ccc8fe0ec5e16678aeb
I20260928 05:27:46.824056 28482 ts_tablet_manager.cc:1939] T 595e0c778f264ccc8fe0ec5e16678aeb P 94021de728a44af1a7f4158a6a75fe46: Deleting consensus metadata
I20260928 05:27:46.824069 28481 raft_consensus.cc:2229] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:46.826427 27967 catalog_manager.cc:4841] TS 94021de728a44af1a7f4158a6a75fe46 (127.8.124.195:43393): tablet 595e0c778f264ccc8fe0ec5e16678aeb (table test-table [id=31f29d61c56347269a2b1be1042d9fcb]) successfully deleted
I20260928 05:27:46.827327 28480 ts_tablet_manager.cc:1918] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:46.827502 28480 log.cc:1196] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-0/wal/wals/595e0c778f264ccc8fe0ec5e16678aeb
I20260928 05:27:46.828029 28480 ts_tablet_manager.cc:1939] T 595e0c778f264ccc8fe0ec5e16678aeb P e22bf12ae1854d039a47d536ebea7ffa: Deleting consensus metadata
I20260928 05:27:46.829180 28481 raft_consensus.cc:2258] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:46.829761 27967 catalog_manager.cc:4841] TS e22bf12ae1854d039a47d536ebea7ffa (127.8.124.193:36165): tablet 595e0c778f264ccc8fe0ec5e16678aeb (table test-table [id=31f29d61c56347269a2b1be1042d9fcb]) successfully deleted
I20260928 05:27:46.831317 28481 ts_tablet_manager.cc:1905] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:46.832608 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.800911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.8.124.193:36165)
0928 05:27:46.801095 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.801103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.806481 (+ 5378us) server_negotiation.cc:407] Connection header received
0928 05:27:46.806684 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.806689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.806770 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.806868 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:46.813280 (+ 6412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.814485 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.816649 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.817372 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.817471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.824734 (+ 7263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:46.824749 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:46.824767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:46.824827 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:46.828886 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.830276 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.830285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.830288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.830436 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.830877 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.830881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.830884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.831329 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:46.831695 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.832052 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.832263 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:27:46.833258 28167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:46.838539 28481 ts_tablet_manager.cc:1918] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:46.838699 28481 log.cc:1196] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/595e0c778f264ccc8fe0ec5e16678aeb
I20260928 05:27:46.839138 28481 ts_tablet_manager.cc:1939] T 595e0c778f264ccc8fe0ec5e16678aeb P 162c77a9402342e2860d70f5372a1126: Deleting consensus metadata
I20260928 05:27:46.840687 27967 catalog_manager.cc:4841] TS 162c77a9402342e2860d70f5372a1126 (127.8.124.194:40199): tablet 595e0c778f264ccc8fe0ec5e16678aeb (table test-table [id=31f29d61c56347269a2b1be1042d9fcb]) successfully deleted
I20260928 05:27:46.970760 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28044
I20260928 05:27:46.985411 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28178
Sep 28 05:27:46 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.194@KRBTEST.COM
I20260928 05:27:46.999997 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:46.986233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.8.124.194:40199)
0928 05:27:46.986411 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:46.986417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:46.986479 (+ 62us) server_negotiation.cc:407] Connection header received
0928 05:27:46.986669 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:46.986674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:46.986757 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:46.986837 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:46.988061 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.988789 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.990068 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:46.990466 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:46.990575 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:46.994710 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:46.994724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:46.994739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:46.994772 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:46.997349 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.998056 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.998063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.998066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.998194 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:46.998542 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:46.998545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:46.998548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:46.998911 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:46.999191 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:46.999484 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 05:27:46.999690 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:47.000571 28301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:47.118687 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28178
I20260928 05:27:47.132899 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28311
Sep 28 05:27:47 dist-test-slave-515x krb5kdc[27938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.195@KRBTEST.COM
I20260928 05:27:47.150849 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:47.133777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.8.124.195:43393)
0928 05:27:47.133994 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:47.134000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:47.134023 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:27:47.134168 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:47.134174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:47.134262 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:47.134384 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:47.135690 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:47.136938 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:47.138378 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:47.139085 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:47.139231 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:47.143494 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:47.143510 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:47.143530 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:47.143578 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:47.147551 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:47.148632 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:47.148643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:47.148646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:47.148868 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:47.149346 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:47.149350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:47.149352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:47.149765 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:47.150016 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:47.150370 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:27:47.150533 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:27:47.151597 28434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:47.273303 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28311
I20260928 05:27:47.288765 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 27951
I20260928 05:27:47.289819 28013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:47.399416 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 27951
2026-09-28T05:27:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4111 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T05:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:47Z Disabled control of system clock
I20260928 05:27:47.471163 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37853
--webserver_interface=127.8.124.254
--webserver_port=0
--builtin_ntp_servers=127.8.124.212:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:47.791640 28491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:47.791934 28491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:47.792049 28491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:47.799293 28491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:47.799407 28491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:47.799448 28491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:47.799525 28491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:47.799580 28491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:47.808619 28491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:37853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:47.809943 28491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:47.811764 28491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:47.822836 28497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:47.822921 28496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:47.822896 28499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:47.823551 28491 server_base.cc:1034] running on GCE node
I20260928 05:27:47.824330 28491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:47.825758 28491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:47.826970 28491 hybrid_clock.cc:648] HybridClock initialized: now 1790573267826919 us; error 41 us; skew 500 ppm
I20260928 05:27:47.827361 28491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:47.829526 28491 webserver.cc:460] Webserver started at http://127.8.124.254:35055/ using document root <none> and password file <none>
I20260928 05:27:47.830235 28491 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:47.830334 28491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:47.830615 28491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:47.833400 28491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "1ba340db917f4c778a0f1efff16e94d3"
format_stamp: "Formatted at 2026-09-28 05:27:47 on dist-test-slave-515x"
server_key: "64a7b0cbb9ed03f10df5c573c2395153"
server_key_iv: "586e95f2fec2e8b3df6bf97308960d80"
server_key_version: "encryptionkey@0"
I20260928 05:27:47.834196 28491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "1ba340db917f4c778a0f1efff16e94d3"
format_stamp: "Formatted at 2026-09-28 05:27:47 on dist-test-slave-515x"
server_key: "64a7b0cbb9ed03f10df5c573c2395153"
server_key_iv: "586e95f2fec2e8b3df6bf97308960d80"
server_key_version: "encryptionkey@0"
I20260928 05:27:47.838954 28491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:27:47.842330 28505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:47.843864 28491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:47.844034 28491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "1ba340db917f4c778a0f1efff16e94d3"
format_stamp: "Formatted at 2026-09-28 05:27:47 on dist-test-slave-515x"
server_key: "64a7b0cbb9ed03f10df5c573c2395153"
server_key_iv: "586e95f2fec2e8b3df6bf97308960d80"
server_key_version: "encryptionkey@0"
I20260928 05:27:47.844271 28491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:47.863540 28491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:47.864519 28491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:47.864778 28491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:47.881752 28491 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:37853
I20260928 05:27:47.881772 28556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:37853 every 8 connection(s)
I20260928 05:27:47.883051 28491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:47.884145 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28491
I20260928 05:27:47.884320 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:47.884855 8691 external_mini_cluster.cc:1428] Setting key 4e8d9ae193c729db27dfef59e8137b79
I20260928 05:27:47.887806 28557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:47.897835 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:47.886812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.8.124.254:37853)
0928 05:27:47.887862 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:47.887877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:47.887930 (+ 53us) server_negotiation.cc:407] Connection header received
0928 05:27:47.889586 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:47.889637 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:47.890107 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:47.890494 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:47.891026 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:47.892365 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:47.893668 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:47.894311 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:47.894468 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:47.895589 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:47.895597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:47.895622 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:47.896183 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:47.896274 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:47.896298 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:47.896398 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:47.896601 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 05:27:47.896883 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":433,"threads_started":1}
I20260928 05:27:47.898272 28557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: Bootstrap starting.
I20260928 05:27:47.901412 28557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:47.902563 28557 log.cc:824] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:47.905655 28557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: No bootstrap required, opened a new log
I20260928 05:27:47.913081 28557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } }
I20260928 05:27:47.913467 28557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:47.913578 28557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba340db917f4c778a0f1efff16e94d3, State: Initialized, Role: FOLLOWER
I20260928 05:27:47.914144 28557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } }
I20260928 05:27:47.914362 28557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:47.914490 28557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:47.914639 28557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:47.916745 28557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } }
I20260928 05:27:47.917279 28557 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba340db917f4c778a0f1efff16e94d3; no voters:
I20260928 05:27:47.917917 28557 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:47.918030 28562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:47.918686 28562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba340db917f4c778a0f1efff16e94d3, State: Running, Role: LEADER
I20260928 05:27:47.919142 28562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } }
I20260928 05:27:47.919894 28557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:47.921648 28564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba340db917f4c778a0f1efff16e94d3. Latest consensus state: current_term: 1 leader_uuid: "1ba340db917f4c778a0f1efff16e94d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } } }
I20260928 05:27:47.922078 28564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:47.922029 28563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba340db917f4c778a0f1efff16e94d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba340db917f4c778a0f1efff16e94d3" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 37853 } } }
I20260928 05:27:47.922299 28563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:47.923787 28570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:47.928104 28570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:47.935681 28570 catalog_manager.cc:1347] Generated new cluster ID: ef6424ffda1c4824aa41af9f3a2f90cc
I20260928 05:27:47.935803 28570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:47.959721 28570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:47.961074 28570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:47.968784 28570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: Generated new TSK 0
I20260928 05:27:47.969578 28570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:47.974474 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.193:0
--local_ip_for_outbound_sockets=127.8.124.193
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--builtin_ntp_servers=127.8.124.212:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:27:48.298297 28581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:48.298668 28581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:48.298813 28581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:48.306023 28581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:48.306130 28581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:48.306243 28581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.193
I20260928 05:27:48.315307 28581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.124.193
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.193
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:48.316548 28581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:48.318540 28581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:48.331758 28586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:48.331717 28587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:48.331674 28589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:48.332682 28581 server_base.cc:1034] running on GCE node
I20260928 05:27:48.333298 28581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:48.334268 28581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:48.335496 28581 hybrid_clock.cc:648] HybridClock initialized: now 1790573268335449 us; error 30 us; skew 500 ppm
I20260928 05:27:48.335812 28581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:48.338080 28581 webserver.cc:460] Webserver started at http://127.8.124.193:40593/ using document root <none> and password file <none>
I20260928 05:27:48.338735 28581 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:48.338817 28581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:48.339075 28581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:48.341971 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data/instance:
uuid: "d0dceb488e794cce89722ef1dcdcbafd"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "aa0864413ff8717b53ea8d7348ac1573"
server_key_iv: "af0737343b6644efb5a99611e934861a"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.342712 28581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance:
uuid: "d0dceb488e794cce89722ef1dcdcbafd"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "aa0864413ff8717b53ea8d7348ac1573"
server_key_iv: "af0737343b6644efb5a99611e934861a"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.347735 28581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 05:27:48.351163 28595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.352468 28581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:27:48.352607 28581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal
uuid: "d0dceb488e794cce89722ef1dcdcbafd"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "aa0864413ff8717b53ea8d7348ac1573"
server_key_iv: "af0737343b6644efb5a99611e934861a"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.352800 28581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:48.370558 28581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:48.371464 28581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:48.371760 28581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:48.372852 28581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:48.374403 28581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:48.374500 28581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.374607 28581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:48.374681 28581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.404421 28581 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.193:38339
I20260928 05:27:48.404479 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.193:38339 every 8 connection(s)
I20260928 05:27:48.405735 28581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/data/info.pb
I20260928 05:27:48.407202 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28581
I20260928 05:27:48.407358 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-0/wal/instance
I20260928 05:27:48.407866 8691 external_mini_cluster.cc:1428] Setting key 80224e6b15d25b5179c0a75962863f59
I20260928 05:27:48.410902 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.194:0
--local_ip_for_outbound_sockets=127.8.124.194
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--builtin_ntp_servers=127.8.124.212:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:48.419121 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:48.408791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.193:51831 (local address 127.8.124.254:37853)
0928 05:27:48.409867 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:48.409872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:48.410897 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 05:27:48.412720 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:48.412726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:48.412818 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:48.412910 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:48.414084 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.414818 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.416391 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.416805 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.416912 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:48.418281 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:48.418304 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:48.418309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:48.418334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:48.418402 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:48.418422 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:48.418520 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:48.418783 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:27:48.418865 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":867,"threads_started":1}
I20260928 05:27:48.420514 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:48.409592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37853 (local address 127.8.124.193:51831)
0928 05:27:48.410742 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:48.410790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:48.412453 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:48.413167 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:48.413176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:48.413279 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:48.413924 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:48.413941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.414966 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.414970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:48.416237 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:48.416249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.417024 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.417030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:48.417270 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:48.417964 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:48.417967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:48.418008 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:48.418090 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:48.418538 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:48.418550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:48.418559 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 05:27:48.418750 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 05:27:48.419149 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":344,"threads_started":1}
I20260928 05:27:48.422608 28708 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37853
I20260928 05:27:48.422945 28708 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:48.424094 28708 heartbeater.cc:511] Master 127.8.124.254:37853 requested a full tablet report, sending...
I20260928 05:27:48.426275 28522 ts_manager.cc:194] Registered new tserver with Master: d0dceb488e794cce89722ef1dcdcbafd (127.8.124.193:38339)
I20260928 05:27:48.427798 28522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.193:51831
W20260928 05:27:48.745693 28712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:48.745985 28712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:48.746110 28712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:48.753459 28712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:48.753575 28712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:48.753744 28712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.194
I20260928 05:27:48.763087 28712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.124.194
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.194
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:48.764420 28712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:48.766387 28712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:48.778728 28717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:48.778707 28718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:48.779076 28720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:48.779517 28712 server_base.cc:1034] running on GCE node
I20260928 05:27:48.780102 28712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:48.781006 28712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:48.782220 28712 hybrid_clock.cc:648] HybridClock initialized: now 1790573268782169 us; error 37 us; skew 500 ppm
I20260928 05:27:48.782543 28712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:48.784750 28712 webserver.cc:460] Webserver started at http://127.8.124.194:35109/ using document root <none> and password file <none>
I20260928 05:27:48.785425 28712 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:48.785523 28712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:48.785876 28712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:48.788728 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data/instance:
uuid: "5da29c698cd14a1c97945179daff189c"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "39c99b84536c302d5a3068c723c66742"
server_key_iv: "793a630df5d838fb4829349e46d84a36"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.789496 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance:
uuid: "5da29c698cd14a1c97945179daff189c"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "39c99b84536c302d5a3068c723c66742"
server_key_iv: "793a630df5d838fb4829349e46d84a36"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.794600 28712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 05:27:48.798076 28726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.799448 28712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:27:48.799629 28712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal
uuid: "5da29c698cd14a1c97945179daff189c"
format_stamp: "Formatted at 2026-09-28 05:27:48 on dist-test-slave-515x"
server_key: "39c99b84536c302d5a3068c723c66742"
server_key_iv: "793a630df5d838fb4829349e46d84a36"
server_key_version: "encryptionkey@0"
I20260928 05:27:48.799836 28712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:48.822912 28712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:48.823768 28712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:48.824047 28712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:48.824996 28712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:48.826515 28712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:48.826604 28712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.826705 28712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:48.826776 28712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:48.854085 28712 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.194:38917
I20260928 05:27:48.854178 28838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.194:38917 every 8 connection(s)
I20260928 05:27:48.855293 28712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/data/info.pb
I20260928 05:27:48.856267 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28712
I20260928 05:27:48.856436 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal/instance
I20260928 05:27:48.857041 8691 external_mini_cluster.cc:1428] Setting key 13e3b1ae79461a07701a42ed09ec4d68
I20260928 05:27:48.860280 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.124.195:0
--local_ip_for_outbound_sockets=127.8.124.195
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--builtin_ntp_servers=127.8.124.212:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:27:48.869969 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:48.858185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:33785 (local address 127.8.124.254:37853)
0928 05:27:48.858347 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:48.858351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:48.860088 (+ 1737us) server_negotiation.cc:407] Connection header received
0928 05:27:48.862464 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:48.862468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:48.862533 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:48.862613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:48.863784 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.864456 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.866718 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.867237 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.867340 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:48.869062 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:48.869068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:48.869072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:48.869094 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:48.869134 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:48.869148 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:48.869285 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:48.869527 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 05:27:48.869648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:27:48.871346 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:48.858851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37853 (local address 127.8.124.194:33785)
0928 05:27:48.859928 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:48.859980 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:48.862183 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:48.862846 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:48.862857 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:48.862972 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:48.863647 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:48.863669 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.864594 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.864599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:48.866570 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:48.866585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:48.867458 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:48.867464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:48.867741 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:48.868482 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:48.868485 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:48.868536 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:48.868633 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:48.869301 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:48.869312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:48.869320 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:27:48.869506 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 05:27:48.870072 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260928 05:27:48.872710 28839 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37853
I20260928 05:27:48.873031 28839 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:48.873939 28839 heartbeater.cc:511] Master 127.8.124.254:37853 requested a full tablet report, sending...
I20260928 05:27:48.875451 28522 ts_manager.cc:194] Registered new tserver with Master: 5da29c698cd14a1c97945179daff189c (127.8.124.194:38917)
I20260928 05:27:48.876252 28522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.194:33785
W20260928 05:27:49.183837 28842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:49.184252 28842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:49.184382 28842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:49.192726 28842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:49.192828 28842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:49.192935 28842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.124.195
I20260928 05:27:49.202196 28842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.124.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.124.195
--webserver_port=0
--tserver_master_addrs=127.8.124.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.124.195
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:49.203477 28842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:49.205253 28842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:49.217705 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:49.217731 28847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:49.217708 28848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:49.218873 28842 server_base.cc:1034] running on GCE node
I20260928 05:27:49.219476 28842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:49.220489 28842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:49.221706 28842 hybrid_clock.cc:648] HybridClock initialized: now 1790573269221642 us; error 55 us; skew 500 ppm
I20260928 05:27:49.222002 28842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:27:49.224246 28842 webserver.cc:460] Webserver started at http://127.8.124.195:41363/ using document root <none> and password file <none>
I20260928 05:27:49.224908 28842 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:49.224995 28842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:49.225257 28842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:49.228171 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data/instance:
uuid: "0533e1b393644e3ba498cdaa8c211dd5"
format_stamp: "Formatted at 2026-09-28 05:27:49 on dist-test-slave-515x"
server_key: "a44694cb5653539839f6b1ddf2af879b"
server_key_iv: "32797d3b85ad5d8b2133c242cb5d0069"
server_key_version: "encryptionkey@0"
I20260928 05:27:49.228932 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance:
uuid: "0533e1b393644e3ba498cdaa8c211dd5"
format_stamp: "Formatted at 2026-09-28 05:27:49 on dist-test-slave-515x"
server_key: "a44694cb5653539839f6b1ddf2af879b"
server_key_iv: "32797d3b85ad5d8b2133c242cb5d0069"
server_key_version: "encryptionkey@0"
I20260928 05:27:49.234110 28842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:27:49.237938 28856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:49.239498 28842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:49.239687 28842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal
uuid: "0533e1b393644e3ba498cdaa8c211dd5"
format_stamp: "Formatted at 2026-09-28 05:27:49 on dist-test-slave-515x"
server_key: "a44694cb5653539839f6b1ddf2af879b"
server_key_iv: "32797d3b85ad5d8b2133c242cb5d0069"
server_key_version: "encryptionkey@0"
I20260928 05:27:49.239916 28842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:49.273737 28842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:49.274577 28842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:49.274859 28842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:49.275825 28842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:27:49.277277 28842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:27:49.277369 28842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:49.277472 28842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:27:49.277540 28842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:49.306604 28842 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.195:44535
I20260928 05:27:49.306670 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.195:44535 every 8 connection(s)
I20260928 05:27:49.307858 28842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/data/info.pb
I20260928 05:27:49.318135 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 28842
I20260928 05:27:49.318307 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal/instance
I20260928 05:27:49.318843 8691 external_mini_cluster.cc:1428] Setting key 8e6cbee17c7979b213dc9bf7d885adb1
I20260928 05:27:49.320247 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.310250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.195:34735 (local address 127.8.124.254:37853)
0928 05:27:49.310452 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.310458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.311908 (+ 1450us) server_negotiation.cc:407] Connection header received
0928 05:27:49.313732 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.313737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.313807 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.313895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:49.314968 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.315687 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.317206 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.317674 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.317790 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.319343 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:49.319350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:49.319354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:49.319378 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:49.319423 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:49.319440 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:49.319525 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.319780 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.319887 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:27:49.321341 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.310865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:37853 (local address 127.8.124.195:34735)
0928 05:27:49.311762 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.311803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.313424 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.314116 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.314124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.314216 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:27:49.314791 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.314817 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.315835 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.315839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.317060 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.317072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.317923 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.317930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.318209 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.318961 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:27:49.318964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:27:49.319015 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:27:49.319119 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:27:49.319548 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:27:49.319558 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:27:49.319567 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.319749 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.320134 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260928 05:27:49.322638 28969 heartbeater.cc:347] Connected to a master server at 127.8.124.254:37853
I20260928 05:27:49.322942 28969 heartbeater.cc:464] Registering TS with master...
I20260928 05:27:49.323665 28969 heartbeater.cc:511] Master 127.8.124.254:37853 requested a full tablet report, sending...
I20260928 05:27:49.325205 28522 ts_manager.cc:194] Registered new tserver with Master: 0533e1b393644e3ba498cdaa8c211dd5 (127.8.124.195:44535)
I20260928 05:27:49.326043 28522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.124.195:34735
I20260928 05:27:49.334158 8691 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:27:49.343894 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.338435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.8.124.254:37853)
0928 05:27:49.338628 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.338633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.339035 (+ 402us) server_negotiation.cc:407] Connection header received
0928 05:27:49.339186 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.339190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.339266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.339349 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:49.339835 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.340548 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.341874 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.342273 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.342367 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.343292 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:49.343314 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:49.343318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:49.343335 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:49.343366 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:49.343378 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:49.343438 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.343538 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.343614 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:49.347546 28522 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:27:49.354600 28522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:27:49.379962 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.366601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.194:38917 (local address 127.0.0.1:41342)
0928 05:27:49.367458 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.367504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.367712 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.369900 (+ 2188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.369909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.369947 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:49.370396 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.370410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.373106 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.373114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.375747 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.375771 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.378299 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.378306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.378566 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.378631 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.378753 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.378989 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":519,"threads_started":1}
I20260928 05:27:49.380448 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.366834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.8.124.194:38917)
0928 05:27:49.368842 (+ 2008us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.368851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.368884 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:27:49.368994 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.369002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.369286 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.370625 (+ 1339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:49.370913 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.372924 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.376390 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.378043 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.379522 (+ 1479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.379792 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.379888 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.380089 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1842,"thread_start_us":1729,"threads_started":1}
I20260928 05:27:49.385401 28774 tablet_service.cc:1467] Processing CreateTablet for tablet a1a9de510e1a4759a5ce42602326e22f (DEFAULT_TABLE table=test-table [id=76582176861a41c7b94e5533e9e3e2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:49.387007 28774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a9de510e1a4759a5ce42602326e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:49.388279 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.367138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:44535 (local address 127.0.0.1:41662)
0928 05:27:49.380282 (+ 13144us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.380304 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.380444 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.381805 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.381811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.381838 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:49.382036 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.382047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.383951 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.383957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.385820 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.385835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.387489 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.387495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.387676 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.387709 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.387815 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.387958 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13009,"mutex_wait_us":711,"thread_start_us":14039,"threads_started":1}
I20260928 05:27:49.388760 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.367578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.8.124.195:44535)
0928 05:27:49.368421 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.368431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.380788 (+ 12357us) server_negotiation.cc:407] Connection header received
0928 05:27:49.380920 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.380929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.381249 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.381449 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:49.382223 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.383800 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.385987 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.387314 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.387437 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.387733 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.387872 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.388035 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":522,"threads_started":1}
I20260928 05:27:49.389153 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.367963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45266)
0928 05:27:49.381653 (+ 13690us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.381675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.381822 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.382840 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.382845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.382869 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:49.383171 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.383180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.384926 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.384931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.386669 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.386680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.388449 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.388455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.388613 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.388644 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.388721 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.388841 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13510,"thread_start_us":13346,"threads_started":1}
I20260928 05:27:49.389554 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.368169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.8.124.193:38339)
0928 05:27:49.379893 (+ 11724us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.379904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.382143 (+ 2239us) server_negotiation.cc:407] Connection header received
0928 05:27:49.382267 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.382274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.382503 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.382714 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:49.383332 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.384739 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.386831 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.387996 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.388112 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.388326 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.388789 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.389126 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11521,"thread_start_us":11380,"threads_started":1}
I20260928 05:27:49.393142 28643 tablet_service.cc:1467] Processing CreateTablet for tablet a1a9de510e1a4759a5ce42602326e22f (DEFAULT_TABLE table=test-table [id=76582176861a41c7b94e5533e9e3e2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:49.393146 28904 tablet_service.cc:1467] Processing CreateTablet for tablet a1a9de510e1a4759a5ce42602326e22f (DEFAULT_TABLE table=test-table [id=76582176861a41c7b94e5533e9e3e2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:27:49.394590 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a9de510e1a4759a5ce42602326e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:49.394582 28904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a9de510e1a4759a5ce42602326e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:49.399657 28987 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Bootstrap starting.
I20260928 05:27:49.402745 28987 tablet_bootstrap.cc:654] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:49.403764 28987 log.cc:824] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:49.416887 28989 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: Bootstrap starting.
I20260928 05:27:49.417850 28990 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Bootstrap starting.
I20260928 05:27:49.422693 28987 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: No bootstrap required, opened a new log
I20260928 05:27:49.422902 28990 tablet_bootstrap.cc:654] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:49.422972 28987 ts_tablet_manager.cc:1397] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.015s
I20260928 05:27:49.423553 28989 tablet_bootstrap.cc:654] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:49.424288 28990 log.cc:824] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:49.424697 28989 log.cc:824] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:49.431257 28708 heartbeater.cc:503] Master 127.8.124.254:37853 was elected leader, sending a full tablet report...
I20260928 05:27:49.430972 28987 raft_consensus.cc:348] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.432229 28987 raft_consensus.cc:374] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:49.432341 28987 raft_consensus.cc:729] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da29c698cd14a1c97945179daff189c, State: Initialized, Role: FOLLOWER
I20260928 05:27:49.432989 28987 consensus_queue.cc:260] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.433280 28989 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: No bootstrap required, opened a new log
I20260928 05:27:49.433465 28990 tablet_bootstrap.cc:492] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: No bootstrap required, opened a new log
I20260928 05:27:49.433564 28989 ts_tablet_manager.cc:1397] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260928 05:27:49.433859 28990 ts_tablet_manager.cc:1397] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260928 05:27:49.435979 28839 heartbeater.cc:503] Master 127.8.124.254:37853 was elected leader, sending a full tablet report...
I20260928 05:27:49.436682 28987 ts_tablet_manager.cc:1428] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260928 05:27:49.441155 28993 raft_consensus.cc:482] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:27:49.441445 28993 raft_consensus.cc:504] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.442850 28989 raft_consensus.cc:348] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.443228 28989 raft_consensus.cc:374] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:49.443320 28989 raft_consensus.cc:729] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dceb488e794cce89722ef1dcdcbafd, State: Initialized, Role: FOLLOWER
I20260928 05:27:49.443495 28990 raft_consensus.cc:348] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.443904 28990 raft_consensus.cc:374] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:49.444002 28990 raft_consensus.cc:729] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0533e1b393644e3ba498cdaa8c211dd5, State: Initialized, Role: FOLLOWER
I20260928 05:27:49.444108 28989 consensus_queue.cc:260] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.444569 28990 consensus_queue.cc:260] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.445739 28969 heartbeater.cc:503] Master 127.8.124.254:37853 was elected leader, sending a full tablet report...
I20260928 05:27:49.447741 28990 ts_tablet_manager.cc:1428] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 05:27:49.451006 28993 leader_election.cc:288] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dceb488e794cce89722ef1dcdcbafd (127.8.124.193:38339), 0533e1b393644e3ba498cdaa8c211dd5 (127.8.124.195:44535)
I20260928 05:27:49.453441 28989 ts_tablet_manager.cc:1428] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260928 05:27:49.457818 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.444017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:39615 (local address 127.8.124.193:38339)
0928 05:27:49.444214 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.444220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.451296 (+ 7076us) server_negotiation.cc:407] Connection header received
0928 05:27:49.451602 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.451608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.451714 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.451782 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:49.452602 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.453684 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.456129 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.456930 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.457073 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.457140 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.457413 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.457537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:27:49.457945 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.450210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.124.194:56053 (local address 127.8.124.195:44535)
0928 05:27:49.450381 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.450386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.451689 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 05:27:49.451921 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.451925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.451993 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.452048 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:27:49.452782 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.453927 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.456241 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.457075 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.457258 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.457335 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.457580 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.457720 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:27:49.458165 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.443478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.8.124.194:39615)
0928 05:27:49.451100 (+ 7622us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.451126 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.451429 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.451963 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.451967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.451999 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:49.452436 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.452451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.454239 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.454243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.455951 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.455965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.457094 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.457100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.457270 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.457301 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.457394 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.457524 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7444,"mutex_wait_us":68,"thread_start_us":7838,"threads_started":1}
I20260928 05:27:49.458200 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.450304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.195:44535 (local address 127.8.124.194:56053)
0928 05:27:49.451599 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.451616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:27:49.451783 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:27:49.452183 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:27:49.452186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:27:49.452204 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:27:49.452625 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.452638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.454040 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.454045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.456054 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:27:49.456065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.457272 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.457279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:27:49.457449 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.457471 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:27:49.457545 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 05:27:49.457768 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":53,"thread_start_us":390,"threads_started":1}
I20260928 05:27:49.459154 28924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a9de510e1a4759a5ce42602326e22f" candidate_uuid: "5da29c698cd14a1c97945179daff189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0533e1b393644e3ba498cdaa8c211dd5" is_pre_election: true
I20260928 05:27:49.459154 28663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a9de510e1a4759a5ce42602326e22f" candidate_uuid: "5da29c698cd14a1c97945179daff189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dceb488e794cce89722ef1dcdcbafd" is_pre_election: true
I20260928 05:27:49.459704 28663 raft_consensus.cc:2454] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da29c698cd14a1c97945179daff189c in term 0.
I20260928 05:27:49.459704 28924 raft_consensus.cc:2454] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da29c698cd14a1c97945179daff189c in term 0.
I20260928 05:27:49.460386 28728 leader_election.cc:302] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da29c698cd14a1c97945179daff189c, d0dceb488e794cce89722ef1dcdcbafd; no voters:
I20260928 05:27:49.460777 28993 raft_consensus.cc:2781] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:27:49.460911 28993 raft_consensus.cc:482] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:27:49.461004 28993 raft_consensus.cc:3037] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:49.463204 28993 raft_consensus.cc:504] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.463846 28993 leader_election.cc:288] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [CANDIDATE]: Term 1 election: Requested vote from peers d0dceb488e794cce89722ef1dcdcbafd (127.8.124.193:38339), 0533e1b393644e3ba498cdaa8c211dd5 (127.8.124.195:44535)
I20260928 05:27:49.464308 28663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a9de510e1a4759a5ce42602326e22f" candidate_uuid: "5da29c698cd14a1c97945179daff189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dceb488e794cce89722ef1dcdcbafd"
I20260928 05:27:49.464308 28924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a9de510e1a4759a5ce42602326e22f" candidate_uuid: "5da29c698cd14a1c97945179daff189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0533e1b393644e3ba498cdaa8c211dd5"
I20260928 05:27:49.464519 28924 raft_consensus.cc:3037] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:49.464519 28663 raft_consensus.cc:3037] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:49.467674 28663 raft_consensus.cc:2454] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da29c698cd14a1c97945179daff189c in term 1.
I20260928 05:27:49.467674 28924 raft_consensus.cc:2454] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da29c698cd14a1c97945179daff189c in term 1.
I20260928 05:27:49.468115 28727 leader_election.cc:302] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0533e1b393644e3ba498cdaa8c211dd5, 5da29c698cd14a1c97945179daff189c, d0dceb488e794cce89722ef1dcdcbafd; no voters:
I20260928 05:27:49.468474 28993 raft_consensus.cc:2781] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:49.468896 28993 raft_consensus.cc:686] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 1 LEADER]: Becoming Leader. State: Replica: 5da29c698cd14a1c97945179daff189c, State: Running, Role: LEADER
I20260928 05:27:49.469416 28993 consensus_queue.cc:237] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } }
I20260928 05:27:49.473835 28522 catalog_manager.cc:5495] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da29c698cd14a1c97945179daff189c (127.8.124.194). New cstate: current_term: 1 leader_uuid: "5da29c698cd14a1c97945179daff189c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da29c698cd14a1c97945179daff189c" member_type: VOTER last_known_addr { host: "127.8.124.194" port: 38917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20260928 05:27:49.491102 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.486031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.8.124.194:38917)
0928 05:27:49.486209 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.486215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.486233 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:27:49.486345 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.486349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.486438 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.486563 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:27:49.486918 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.487636 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.489008 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.489429 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.489534 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.490037 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:27:49.490546 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:27:49.490674 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.490771 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.490848 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:27:49.498975 28663 raft_consensus.cc:1261] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd [term 1 FOLLOWER]: Refusing update from remote peer 5da29c698cd14a1c97945179daff189c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:49.498976 28924 raft_consensus.cc:1261] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 1 FOLLOWER]: Refusing update from remote peer 5da29c698cd14a1c97945179daff189c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:27:49.499979 28993 consensus_queue.cc:1035] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0533e1b393644e3ba498cdaa8c211dd5" member_type: VOTER last_known_addr { host: "127.8.124.195" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:49.500248 28998 consensus_queue.cc:1035] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dceb488e794cce89722ef1dcdcbafd" member_type: VOTER last_known_addr { host: "127.8.124.193" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:27:49.511379 29001 mvcc.cc:205] Tried to move back new op lower bound from 7334188111857811456 to 7334188111750782976. Current Snapshot: MvccSnapshot[applied={T|T < 7334188111857811456}]
I20260928 05:27:49.513574 29002 mvcc.cc:205] Tried to move back new op lower bound from 7334188111857811456 to 7334188111750782976. Current Snapshot: MvccSnapshot[applied={T|T < 7334188111857811456}]
I20260928 05:27:49.524981 28522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:49.525200 28522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:27:49.528716 28522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba340db917f4c778a0f1efff16e94d3: Sending DeleteTablet for 3 replicas of tablet a1a9de510e1a4759a5ce42602326e22f
I20260928 05:27:49.530176 28904 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a9de510e1a4759a5ce42602326e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:49 UTC) from {username='slave'} at 127.0.0.1:41662
I20260928 05:27:49.530315 28774 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a9de510e1a4759a5ce42602326e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:49 UTC) from {username='slave'} at 127.0.0.1:41342
I20260928 05:27:49.530993 28643 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a9de510e1a4759a5ce42602326e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:27:49 UTC) from {username='slave'} at 127.0.0.1:45266
I20260928 05:27:49.531704 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28581
I20260928 05:27:49.551017 29013 tablet_replica.cc:330] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: stopping tablet replica
I20260928 05:27:49.551611 29013 raft_consensus.cc:2229] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:27:49.552464 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.533220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.8.124.193:38339)
0928 05:27:49.533410 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.533416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.543259 (+ 9843us) server_negotiation.cc:407] Connection header received
0928 05:27:49.543791 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.543798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.543948 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.544044 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:49.544865 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.546169 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.548332 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.549061 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.549160 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.550774 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:49.550787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:49.550806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:49.551579 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:49.551645 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:49.551672 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:49.551810 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.552008 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.552126 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:27:49.552474 29013 raft_consensus.cc:2258] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:49.553642 28703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:49.554222 29012 tablet_replica.cc:330] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: stopping tablet replica
I20260928 05:27:49.554663 29012 raft_consensus.cc:2229] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:27:49.555168 29012 raft_consensus.cc:2258] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:27:49.557289 29012 ts_tablet_manager.cc:1905] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:49.564289 29012 ts_tablet_manager.cc:1918] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:49.564440 29012 log.cc:1196] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-2/wal/wals/a1a9de510e1a4759a5ce42602326e22f
I20260928 05:27:49.564940 29012 ts_tablet_manager.cc:1939] T a1a9de510e1a4759a5ce42602326e22f P 0533e1b393644e3ba498cdaa8c211dd5: Deleting consensus metadata
I20260928 05:27:49.566042 29013 ts_tablet_manager.cc:1905] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:27:49.566795 28506 catalog_manager.cc:4841] TS 0533e1b393644e3ba498cdaa8c211dd5 (127.8.124.195:44535): tablet a1a9de510e1a4759a5ce42602326e22f (table test-table [id=76582176861a41c7b94e5533e9e3e2ec]) successfully deleted
I20260928 05:27:49.573577 29013 ts_tablet_manager.cc:1918] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:27:49.573793 29013 log.cc:1196] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Deleting WAL directory at /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572965871275-8691-0/minicluster-data/ts-1/wal/wals/a1a9de510e1a4759a5ce42602326e22f
I20260928 05:27:49.574348 29013 ts_tablet_manager.cc:1939] T a1a9de510e1a4759a5ce42602326e22f P 5da29c698cd14a1c97945179daff189c: Deleting consensus metadata
I20260928 05:27:49.576182 28507 catalog_manager.cc:4841] TS 5da29c698cd14a1c97945179daff189c (127.8.124.194:38917): tablet a1a9de510e1a4759a5ce42602326e22f (table test-table [id=76582176861a41c7b94e5533e9e3e2ec]) successfully deleted
I20260928 05:27:49.644881 29014 tablet_replica.cc:330] T a1a9de510e1a4759a5ce42602326e22f P d0dceb488e794cce89722ef1dcdcbafd: stopping tablet replica
I20260928 05:27:49.645226 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28581
I20260928 05:27:49.666687 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28712
W20260928 05:27:49.666807 28507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.124.193:38339 (error 108)
I20260928 05:27:49.669159 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.668742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45278)
0928 05:27:49.668922 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.669020 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:27:49.669478 28507 catalog_manager.cc:4584] TS d0dceb488e794cce89722ef1dcdcbafd (127.8.124.193:38339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1a9de510e1a4759a5ce42602326e22f: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
I20260928 05:27:49.674476 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.667510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.8.124.194:38917)
0928 05:27:49.667699 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.667705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.667725 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:27:49.667908 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.667914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.668051 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.668172 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:49.668572 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.669745 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.671024 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.671671 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.671793 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.672765 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:49.672780 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:49.672798 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:49.673595 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:49.673681 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:49.673714 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:49.673833 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.673990 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.674092 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:27:49.675213 28834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:49.723026 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.722560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45284)
0928 05:27:49.722761 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.722861 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:27:49.724048 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.723708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45298)
0928 05:27:49.723840 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.723927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 05:27:49.784500 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28712
I20260928 05:27:49.786957 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.784322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45306)
0928 05:27:49.786662 (+ 2340us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.786774 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2178}
I20260928 05:27:49.788164 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.787771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45318)
0928 05:27:49.787953 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.788028 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:27:49.800516 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28842
I20260928 05:27:49.807359 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.801199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.8.124.195:44535)
0928 05:27:49.801401 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:49.801407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:49.801499 (+ 92us) server_negotiation.cc:407] Connection header received
0928 05:27:49.801735 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:49.801739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:49.801819 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:49.801902 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:49.802314 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.803040 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.804306 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:49.804717 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:49.804822 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:49.805897 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:49.805906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:49.805922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:49.806557 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:49.806605 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:49.806627 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:49.806815 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:49.806910 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 05:27:49.807026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:27:49.807984 28964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:49.868673 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.868208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45320)
0928 05:27:49.868420 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.868506 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 05:27:49.869737 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:49.869333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.193:38339 (local address 127.0.0.1:45332)
0928 05:27:49.869516 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:27:49.869587 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.193:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 05:27:49.895700 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28842
I20260928 05:27:49.909188 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 28491
I20260928 05:27:49.910256 28552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:50.016466 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 28491
2026-09-28T05:27:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2616 ms)
[----------] 6 tests from EncryptionPolicyTest (20501 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:50 dist-test-slave-515x krb5kdc[29023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:50 dist-test-slave-515x krb5kdc[29023](info): set up 2 sockets
Sep 28 05:27:50 dist-test-slave-515x krb5kdc[29023](info): commencing operation
krb5kdc: starting...
W20260928 05:27:51.041985 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:51 dist-test-slave-515x krb5kdc[29023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:27:51.296635 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:51.625418 29036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:51.625753 29036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:51.625891 29036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:51.633741 29036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:51.633867 29036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:51.633935 29036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:51.634027 29036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:51.634076 29036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:51.644173 29036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:51.645506 29036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:51.646824 29036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:51.660166 29041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:51.660188 29044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:51.660130 29042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:51.660913 29036 server_base.cc:1034] running on GCE node
I20260928 05:27:51.661751 29036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:51.663149 29036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:51.664383 29036 hybrid_clock.cc:648] HybridClock initialized: now 1790573271664328 us; error 46 us; skew 500 ppm
I20260928 05:27:51.664757 29036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:51 dist-test-slave-515x krb5kdc[29023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:51.667865 29036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:27:51.669358 29036 webserver.cc:460] Webserver started at http://127.0.0.1:39843/ using document root <none> and password file <none>
I20260928 05:27:51.670137 29036 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:51.670248 29036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:51.670493 29036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:51.673372 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "c20559b8416440ef893d6f3ddb683c02"
format_stamp: "Formatted at 2026-09-28 05:27:51 on dist-test-slave-515x"
server_key: "c010362c8b0b2d18d4484c507ba1635c"
server_key_iv: "cb1c9a82d31bd68658fe8ae52c682fd5"
server_key_version: "encryptionkey@0"
I20260928 05:27:51.674141 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "c20559b8416440ef893d6f3ddb683c02"
format_stamp: "Formatted at 2026-09-28 05:27:51 on dist-test-slave-515x"
server_key: "c010362c8b0b2d18d4484c507ba1635c"
server_key_iv: "cb1c9a82d31bd68658fe8ae52c682fd5"
server_key_version: "encryptionkey@0"
I20260928 05:27:51.679219 29036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:27:51.682729 29051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:51.684177 29036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:27:51.684330 29036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "c20559b8416440ef893d6f3ddb683c02"
format_stamp: "Formatted at 2026-09-28 05:27:51 on dist-test-slave-515x"
server_key: "c010362c8b0b2d18d4484c507ba1635c"
server_key_iv: "cb1c9a82d31bd68658fe8ae52c682fd5"
server_key_version: "encryptionkey@0"
I20260928 05:27:51.684499 29036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:51.696749 29036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:51.700896 29036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:51.701121 29036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:51.718685 29036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34399
I20260928 05:27:51.718734 29102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34399 every 8 connection(s)
I20260928 05:27:51.720022 29036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:51.721129 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29036
I20260928 05:27:51.721304 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:51.722098 8691 external_mini_cluster.cc:1428] Setting key ea3a1c06a1210732fe62667a518b4976
I20260928 05:27:51.724619 29103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:27:51 dist-test-slave-515x krb5kdc[29023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:27:51.737299 29103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02: Bootstrap starting.
I20260928 05:27:51.740795 29103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:51.742089 29103 log.cc:824] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:51.745253 29103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02: No bootstrap required, opened a new log
I20260928 05:27:51.745720 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:51.724124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60631 (local address 127.0.0.1:34399)
0928 05:27:51.724914 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:51.724925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:51.724967 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:27:51.726584 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:51.726604 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:51.726999 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:51.727333 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:51.729047 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:51.730434 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:51.732120 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:51.732777 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:51.732898 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:51.737838 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:51.737846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:51.737861 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:51.737904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:51.740974 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:51.741980 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:51.741987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:51.741993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:51.742172 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:51.742709 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:51.742715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:51.742718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:51.743296 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:51.743609 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:51.744029 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 05:27:51.744371 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":333,"threads_started":1}
I20260928 05:27:51.749857 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:27:51.753453 29103 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } }
I20260928 05:27:51.753844 29103 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:51.753940 29103 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20559b8416440ef893d6f3ddb683c02, State: Initialized, Role: FOLLOWER
I20260928 05:27:51.754525 29103 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } }
I20260928 05:27:51.754743 29103 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:51.754873 29103 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:51.755023 29103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:51.757529 29103 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } }
I20260928 05:27:51.758145 29103 leader_election.cc:302] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20559b8416440ef893d6f3ddb683c02; no voters:
I20260928 05:27:51.758764 29103 leader_election.cc:288] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:51.758867 29113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:51.760854 29113 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [term 1 LEADER]: Becoming Leader. State: Replica: c20559b8416440ef893d6f3ddb683c02, State: Running, Role: LEADER
I20260928 05:27:51.761071 29103 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:51.761402 29113 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } }
I20260928 05:27:51.765086 29114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20559b8416440ef893d6f3ddb683c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } } }
I20260928 05:27:51.765087 29115 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20559b8416440ef893d6f3ddb683c02. Latest consensus state: current_term: 1 leader_uuid: "c20559b8416440ef893d6f3ddb683c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20559b8416440ef893d6f3ddb683c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34399 } } }
I20260928 05:27:51.765733 29115 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:51.765573 29114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:51.768504 29121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:51.769945 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:51.754284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48833 (local address 127.0.0.1:34399)
0928 05:27:51.754453 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:51.754459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:51.755241 (+ 782us) server_negotiation.cc:407] Connection header received
0928 05:27:51.755366 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:51.755371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:51.755445 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:51.755548 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:51.756836 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:51.757933 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:51.759243 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:51.759855 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:51.759929 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:51.762609 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:51.762617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:51.762621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:51.762662 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:51.766553 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:51.767466 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:51.767470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:51.767472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:51.767631 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:51.768147 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:51.768152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:51.768155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:51.768544 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:51.768866 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:51.769414 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 05:27:51.769582 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:27:51.774210 29121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:51.782737 29121 catalog_manager.cc:1347] Generated new cluster ID: 606d41adbcf040bb8ce61180e4f4ff53
I20260928 05:27:51.782852 29121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:51.818506 29121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:51.820415 29121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:51.837458 29121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20559b8416440ef893d6f3ddb683c02: Generated new TSK 0
I20260928 05:27:51.838580 29121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:51.846380 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29036
I20260928 05:27:51.847797 29098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:51.940176 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29036
W20260928 05:27:51.954311 29021 connection.cc:535] client connection to 127.0.0.1:34399 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34399): Connection reset by peer (error 104)
2026-09-28T05:27:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:52 dist-test-slave-515x krb5kdc[29135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:52 dist-test-slave-515x krb5kdc[29135](info): set up 2 sockets
Sep 28 05:27:52 dist-test-slave-515x krb5kdc[29135](info): commencing operation
krb5kdc: starting...
W20260928 05:27:52.965881 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:53 dist-test-slave-515x krb5kdc[29135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:27:53.205768 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:53.537336 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:53.537619 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:53.537760 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:53.544902 29148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:53.545011 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:53.545082 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:53.545158 29148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:53.545215 29148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:53.554965 29148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:53.556190 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:53.557525 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:53.569056 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:53.569085 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:53.569006 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:53.569907 29148 server_base.cc:1034] running on GCE node
I20260928 05:27:53.570624 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:53.571985 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:53.573197 29148 hybrid_clock.cc:648] HybridClock initialized: now 1790573273573135 us; error 41 us; skew 500 ppm
I20260928 05:27:53.573513 29148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:53 dist-test-slave-515x krb5kdc[29135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:53.576361 29148 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:27:53.577757 29148 webserver.cc:460] Webserver started at http://127.0.0.1:44585/ using document root <none> and password file <none>
I20260928 05:27:53.578400 29148 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:53.578510 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:53.578788 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:53.581622 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "ae1b9c4ebda04e5f9d92748b0eac8152"
format_stamp: "Formatted at 2026-09-28 05:27:53 on dist-test-slave-515x"
server_key: "78456758daa385266a75df6d5c7e529d"
server_key_iv: "7e566a2717aed3c738fe9b2fdcb8cfb7"
server_key_version: "encryptionkey@0"
I20260928 05:27:53.582355 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "ae1b9c4ebda04e5f9d92748b0eac8152"
format_stamp: "Formatted at 2026-09-28 05:27:53 on dist-test-slave-515x"
server_key: "78456758daa385266a75df6d5c7e529d"
server_key_iv: "7e566a2717aed3c738fe9b2fdcb8cfb7"
server_key_version: "encryptionkey@0"
I20260928 05:27:53.587405 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:27:53.590926 29163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:53.592394 29148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:27:53.592569 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "ae1b9c4ebda04e5f9d92748b0eac8152"
format_stamp: "Formatted at 2026-09-28 05:27:53 on dist-test-slave-515x"
server_key: "78456758daa385266a75df6d5c7e529d"
server_key_iv: "7e566a2717aed3c738fe9b2fdcb8cfb7"
server_key_version: "encryptionkey@0"
I20260928 05:27:53.592764 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:53.652246 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:53.656302 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:53.656559 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:53.673506 29148 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38017
I20260928 05:27:53.673533 29214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38017 every 8 connection(s)
I20260928 05:27:53.674850 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:53.678527 29215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:53.679319 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29148
I20260928 05:27:53.679505 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:53.680056 8691 external_mini_cluster.cc:1428] Setting key 526f4d72f089af0c405ff547765478b7
I20260928 05:27:53.691959 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152: Bootstrap starting.
I20260928 05:27:53.693531 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:53.681959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44071 (local address 127.0.0.1:38017)
0928 05:27:53.682722 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:53.682734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:53.682777 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:27:53.684793 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:53.684819 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:53.685246 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:53.685577 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:53.686187 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:53.687418 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:53.689283 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:53.690012 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:53.690182 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:53.691357 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:53.691364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:53.691378 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:53.691889 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:53.691969 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:53.691994 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:53.692176 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:53.692374 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 05:27:53.692549 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":340,"threads_started":1}
I20260928 05:27:53.695220 29215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:53.696455 29215 log.cc:824] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:53.697707 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:27:53.699246 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152: No bootstrap required, opened a new log
I20260928 05:27:53.708185 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:53.701878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44975 (local address 127.0.0.1:38017)
0928 05:27:53.702062 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:53.702068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:53.702411 (+ 343us) server_negotiation.cc:407] Connection header received
0928 05:27:53.702563 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:53.702569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:53.702648 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:53.702771 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:53.703189 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:53.704343 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:53.705636 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:53.706337 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:53.706462 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:53.707427 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:53.707432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:53.707437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:53.707456 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:53.707518 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:53.707538 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:53.707643 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:53.707771 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 05:27:53.707822 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:27:53.707921 29215 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260928 05:27:53.708438 29215 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:53.708544 29215 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b9c4ebda04e5f9d92748b0eac8152, State: Initialized, Role: FOLLOWER
I20260928 05:27:53.709107 29215 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260928 05:27:53.709357 29215 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:53.709512 29215 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:53.709707 29215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:53.711798 29215 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260928 05:27:53.712266 29215 leader_election.cc:302] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae1b9c4ebda04e5f9d92748b0eac8152; no voters:
I20260928 05:27:53.712797 29215 leader_election.cc:288] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:53.712939 29225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:53.713966 29225 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [term 1 LEADER]: Becoming Leader. State: Replica: ae1b9c4ebda04e5f9d92748b0eac8152, State: Running, Role: LEADER
I20260928 05:27:53.714504 29225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260928 05:27:53.714848 29215 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:53.717396 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } } }
I20260928 05:27:53.717862 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:53.717711 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae1b9c4ebda04e5f9d92748b0eac8152. Latest consensus state: current_term: 1 leader_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1b9c4ebda04e5f9d92748b0eac8152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } } }
I20260928 05:27:53.718056 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:53.719551 29234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:53.723804 29234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:53.736127 29234 catalog_manager.cc:1347] Generated new cluster ID: 3b090775dbcd41c1b527daf5caf016c9
I20260928 05:27:53.736253 29234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:53.759011 29234 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:53.760280 29234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:53.765391 29234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae1b9c4ebda04e5f9d92748b0eac8152: Generated new TSK 0
I20260928 05:27:53.766198 29234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:53.779908 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29148
W20260928 05:27:53.781090 29210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44071
W20260928 05:27:53.781843 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29148: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:27:53.782016 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29148
W20260928 05:27:53.795680 29133 connection.cc:535] client connection to 127.0.0.1:38017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38017): Connection reset by peer (error 104)
2026-09-28T05:27:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1840 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:53 dist-test-slave-515x krb5kdc[29246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:53 dist-test-slave-515x krb5kdc[29246](info): set up 2 sockets
Sep 28 05:27:53 dist-test-slave-515x krb5kdc[29246](info): commencing operation
krb5kdc: starting...
W20260928 05:27:54.805810 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:54 dist-test-slave-515x krb5kdc[29246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:27:55.056459 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:55.407956 29259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:55.408308 29259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:55.408458 29259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:55.416047 29259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:55.416193 29259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:55.416280 29259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:55.416363 29259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:55.416422 29259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:55.426811 29259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:55.428113 29259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:55.429445 29259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:55.441104 29265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:55.441165 29267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:55.441115 29264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:55.441586 29259 server_base.cc:1034] running on GCE node
I20260928 05:27:55.442577 29259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:55.444010 29259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:55.445221 29259 hybrid_clock.cc:648] HybridClock initialized: now 1790573275445170 us; error 46 us; skew 500 ppm
I20260928 05:27:55.445578 29259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:55 dist-test-slave-515x krb5kdc[29246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:55.453279 29259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:27:55.455020 29259 webserver.cc:460] Webserver started at http://127.0.0.1:46721/ using document root <none> and password file <none>
I20260928 05:27:55.455765 29259 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:55.455897 29259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:55.456203 29259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:55.459368 29259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "b4c927208ec546538e6f79b1e41a3dfb"
format_stamp: "Formatted at 2026-09-28 05:27:55 on dist-test-slave-515x"
server_key: "30cb4db40f29d386e9f53979d368506f"
server_key_iv: "34791a4cb8590b298bda6e94bf58694e"
server_key_version: "encryptionkey@0"
I20260928 05:27:55.460144 29259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "b4c927208ec546538e6f79b1e41a3dfb"
format_stamp: "Formatted at 2026-09-28 05:27:55 on dist-test-slave-515x"
server_key: "30cb4db40f29d386e9f53979d368506f"
server_key_iv: "34791a4cb8590b298bda6e94bf58694e"
server_key_version: "encryptionkey@0"
I20260928 05:27:55.465583 29259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:27:55.469403 29274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:55.470964 29259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:27:55.471133 29259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "b4c927208ec546538e6f79b1e41a3dfb"
format_stamp: "Formatted at 2026-09-28 05:27:55 on dist-test-slave-515x"
server_key: "30cb4db40f29d386e9f53979d368506f"
server_key_iv: "34791a4cb8590b298bda6e94bf58694e"
server_key_version: "encryptionkey@0"
I20260928 05:27:55.471324 29259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:55.476986 29259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:55.481101 29259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:55.481383 29259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:55.499763 29259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35905
I20260928 05:27:55.499809 29325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35905 every 8 connection(s)
I20260928 05:27:55.501176 29259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:55.501833 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29259
I20260928 05:27:55.502031 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:55.502573 8691 external_mini_cluster.cc:1428] Setting key 1ae1679e2503f9acc3df1353f9427a45
I20260928 05:27:55.506479 29326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:55.510612 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:55.504535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32955 (local address 127.0.0.1:35905)
0928 05:27:55.505294 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:55.505307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:55.505351 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:27:55.507086 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:55.507111 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:55.507470 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:55.507799 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:55.508907 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:55.508913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:55.508929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:55.508943 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:55.509029 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:55.509052 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:55.509257 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:55.509423 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 05:27:55.509678 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":321,"threads_started":1}
I20260928 05:27:55.514670 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:27:55.520576 29326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb: Bootstrap starting.
I20260928 05:27:55.523043 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:55.520319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34525 (local address 127.0.0.1:35905)
0928 05:27:55.520556 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:55.520564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:55.520991 (+ 427us) server_negotiation.cc:407] Connection header received
0928 05:27:55.521230 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:55.521234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:55.521314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:55.521407 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:55.522406 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:55.522410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:55.522413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:55.522431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:55.522486 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:55.522501 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:55.522604 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:55.522710 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 05:27:55.522771 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:27:55.524734 29326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:55.525839 29326 log.cc:824] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:55.528561 29326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb: No bootstrap required, opened a new log
I20260928 05:27:55.536026 29326 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 05:27:55.536401 29326 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:55.536492 29326 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c927208ec546538e6f79b1e41a3dfb, State: Initialized, Role: FOLLOWER
I20260928 05:27:55.537058 29326 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 05:27:55.537276 29326 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:55.537395 29326 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:55.537541 29326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:55.539727 29326 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 05:27:55.540253 29326 leader_election.cc:302] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c927208ec546538e6f79b1e41a3dfb; no voters:
I20260928 05:27:55.540823 29326 leader_election.cc:288] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:55.540958 29336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:55.541635 29336 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [term 1 LEADER]: Becoming Leader. State: Replica: b4c927208ec546538e6f79b1e41a3dfb, State: Running, Role: LEADER
I20260928 05:27:55.542228 29336 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260928 05:27:55.542910 29326 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:55.545964 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c927208ec546538e6f79b1e41a3dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260928 05:27:55.546391 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:55.545569 29338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c927208ec546538e6f79b1e41a3dfb. Latest consensus state: current_term: 1 leader_uuid: "b4c927208ec546538e6f79b1e41a3dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c927208ec546538e6f79b1e41a3dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260928 05:27:55.549901 29338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:55.550626 29345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:55.556510 29345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:55.565917 29345 catalog_manager.cc:1347] Generated new cluster ID: af016266ec4f4e5f9c941fe5ae5dd389
I20260928 05:27:55.566045 29345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:55.579288 29345 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:55.579838 29345 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:55.589931 29345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c927208ec546538e6f79b1e41a3dfb: Generated new TSK 0
I20260928 05:27:55.591306 29345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:55.596055 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29259
W20260928 05:27:55.597218 29321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32955
W20260928 05:27:55.597939 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29259: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:27:55.598100 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29259
2026-09-28T05:27:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:55 dist-test-slave-515x krb5kdc[29357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:55 dist-test-slave-515x krb5kdc[29357](info): set up 2 sockets
Sep 28 05:27:55 dist-test-slave-515x krb5kdc[29357](info): commencing operation
krb5kdc: starting...
W20260928 05:27:56.613781 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:56 dist-test-slave-515x krb5kdc[29357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:27:56.861060 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:57.192022 29370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:57.192305 29370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:57.192399 29370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:57.199803 29370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:57.199925 29370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:57.199971 29370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:57.200030 29370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:57.200073 29370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:57.210078 29370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:57.211313 29370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:57.212564 29370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:57.224078 29375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:57.224226 29378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:57.224264 29376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:57.224792 29370 server_base.cc:1034] running on GCE node
I20260928 05:27:57.225523 29370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:57.226927 29370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:57.228130 29370 hybrid_clock.cc:648] HybridClock initialized: now 1790573277228049 us; error 68 us; skew 500 ppm
I20260928 05:27:57.228475 29370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:57 dist-test-slave-515x krb5kdc[29357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:57.231567 29370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:27:57.233007 29370 webserver.cc:460] Webserver started at http://127.0.0.1:39427/ using document root <none> and password file <none>
I20260928 05:27:57.233695 29370 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:57.233803 29370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:57.234082 29370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:57.236922 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "88c5be49cbc3423683a060c0efc26290"
format_stamp: "Formatted at 2026-09-28 05:27:57 on dist-test-slave-515x"
server_key: "a393330abcecde2ad6373625e6c2e5db"
server_key_iv: "51ce66dca7f63414746a4ccec45d9e63"
server_key_version: "encryptionkey@0"
I20260928 05:27:57.237663 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "88c5be49cbc3423683a060c0efc26290"
format_stamp: "Formatted at 2026-09-28 05:27:57 on dist-test-slave-515x"
server_key: "a393330abcecde2ad6373625e6c2e5db"
server_key_iv: "51ce66dca7f63414746a4ccec45d9e63"
server_key_version: "encryptionkey@0"
I20260928 05:27:57.242583 29370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:27:57.245942 29385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:57.247246 29370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:27:57.247402 29370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "88c5be49cbc3423683a060c0efc26290"
format_stamp: "Formatted at 2026-09-28 05:27:57 on dist-test-slave-515x"
server_key: "a393330abcecde2ad6373625e6c2e5db"
server_key_iv: "51ce66dca7f63414746a4ccec45d9e63"
server_key_version: "encryptionkey@0"
I20260928 05:27:57.247575 29370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:57.270934 29370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:57.275281 29370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:57.275549 29370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:57.292161 29370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35923
I20260928 05:27:57.292235 29436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35923 every 8 connection(s)
I20260928 05:27:57.293494 29370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:57.294605 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29370
I20260928 05:27:57.294775 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:57.295305 8691 external_mini_cluster.cc:1428] Setting key 89b9192096c6f400fc1d1c0fcce8cff1
I20260928 05:27:57.297914 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:27:57 dist-test-slave-515x krb5kdc[29357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:27:57.310719 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290: Bootstrap starting.
I20260928 05:27:57.313858 29437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:57.315052 29437 log.cc:824] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:57.317827 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:57.297014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57097 (local address 127.0.0.1:35923)
0928 05:27:57.297766 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:57.297779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:57.297820 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:27:57.299425 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:57.299445 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:57.299861 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:57.300179 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:57.301929 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:57.303346 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:57.305066 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:57.305761 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:57.305880 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:57.310566 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:57.310573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:57.310588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:57.310643 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:57.313559 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:57.314404 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:57.314411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:57.314418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:57.314569 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:57.315043 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:57.315048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:57.315050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:57.315704 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:57.316006 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:57.316428 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 05:27:57.316815 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":330,"threads_started":1}
I20260928 05:27:57.318110 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290: No bootstrap required, opened a new log
I20260928 05:27:57.321780 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:27:57.327796 29437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260928 05:27:57.328260 29437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:57.328368 29437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c5be49cbc3423683a060c0efc26290, State: Initialized, Role: FOLLOWER
I20260928 05:27:57.329082 29437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260928 05:27:57.329356 29437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:57.329504 29437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:57.329749 29437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:57.332744 29437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260928 05:27:57.333249 29437 leader_election.cc:302] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c5be49cbc3423683a060c0efc26290; no voters:
I20260928 05:27:57.333938 29437 leader_election.cc:288] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:57.334064 29447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:57.335896 29437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:57.336365 29447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [term 1 LEADER]: Becoming Leader. State: Replica: 88c5be49cbc3423683a060c0efc26290, State: Running, Role: LEADER
I20260928 05:27:57.336948 29447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260928 05:27:57.339964 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c5be49cbc3423683a060c0efc26290. Latest consensus state: current_term: 1 leader_uuid: "88c5be49cbc3423683a060c0efc26290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } } }
I20260928 05:27:57.340354 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:57.342290 29448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c5be49cbc3423683a060c0efc26290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5be49cbc3423683a060c0efc26290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } } }
I20260928 05:27:57.342680 29448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:57.344996 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:57.326974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36375 (local address 127.0.0.1:35923)
0928 05:27:57.327154 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:57.327169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:57.327644 (+ 475us) server_negotiation.cc:407] Connection header received
0928 05:27:57.327808 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:57.327812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:57.327876 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:57.327959 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:57.329805 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:57.330637 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:57.332421 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:57.332895 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:57.333023 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:57.336155 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:57.336170 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:57.336175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:27:57.336219 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:57.340409 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:57.341436 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:57.341442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:57.341447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:57.341692 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:27:57.342205 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:27:57.342210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:27:57.342214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:27:57.342686 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:57.344138 (+ 1452us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:57.344319 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 05:27:57.344564 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260928 05:27:57.345197 29455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:57.350530 29455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:57.359593 29455 catalog_manager.cc:1347] Generated new cluster ID: 8817c3025cec41fe8e665d8d7dae18d0
I20260928 05:27:57.359706 29455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:57.370512 29455 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:57.371762 29455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:57.381750 29455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c5be49cbc3423683a060c0efc26290: Generated new TSK 0
I20260928 05:27:57.382536 29455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:57.394112 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29370
I20260928 05:27:57.395113 29432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:27:57.484076 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29370
2026-09-28T05:27:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:57 dist-test-slave-515x krb5kdc[29469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:57 dist-test-slave-515x krb5kdc[29469](info): set up 2 sockets
Sep 28 05:27:57 dist-test-slave-515x krb5kdc[29469](info): commencing operation
krb5kdc: starting...
W20260928 05:27:58.498004 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:27:58 dist-test-slave-515x krb5kdc[29469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:27:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:27:58.742628 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:27:59.067126 29482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:27:59.067438 29482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:27:59.067580 29482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:27:59.075043 29482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:27:59.075174 29482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:27:59.075255 29482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:27:59.075337 29482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:27:59.075397 29482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:27:59.085335 29482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:27:59.086629 29482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:27:59.087913 29482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:27:59.098874 29487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:59.098937 29490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:27:59.098976 29488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:27:59.100009 29482 server_base.cc:1034] running on GCE node
I20260928 05:27:59.100785 29482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:27:59.102227 29482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:27:59.103423 29482 hybrid_clock.cc:648] HybridClock initialized: now 1790573279103381 us; error 30 us; skew 500 ppm
I20260928 05:27:59.103765 29482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:27:59 dist-test-slave-515x krb5kdc[29469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:27:59.107208 29482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:27:59.108723 29482 webserver.cc:460] Webserver started at http://127.0.0.1:40231/ using document root <none> and password file <none>
I20260928 05:27:59.109423 29482 fs_manager.cc:359] Metadata directory not provided
I20260928 05:27:59.109542 29482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:27:59.109879 29482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:27:59.112890 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7"
format_stamp: "Formatted at 2026-09-28 05:27:59 on dist-test-slave-515x"
server_key: "dbdf98c2e894fe9580185187876cc658"
server_key_iv: "1676ec3d088b8b7d83cb678c51ff7ea3"
server_key_version: "encryptionkey@0"
I20260928 05:27:59.113765 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7"
format_stamp: "Formatted at 2026-09-28 05:27:59 on dist-test-slave-515x"
server_key: "dbdf98c2e894fe9580185187876cc658"
server_key_iv: "1676ec3d088b8b7d83cb678c51ff7ea3"
server_key_version: "encryptionkey@0"
I20260928 05:27:59.119194 29482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 05:27:59.122910 29497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:27:59.124303 29482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:27:59.124477 29482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7"
format_stamp: "Formatted at 2026-09-28 05:27:59 on dist-test-slave-515x"
server_key: "dbdf98c2e894fe9580185187876cc658"
server_key_iv: "1676ec3d088b8b7d83cb678c51ff7ea3"
server_key_version: "encryptionkey@0"
I20260928 05:27:59.124661 29482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:27:59.141141 29482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:27:59.145462 29482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:27:59.145749 29482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:27:59.163019 29482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44149
I20260928 05:27:59.163049 29548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44149 every 8 connection(s)
I20260928 05:27:59.164350 29482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:27:59.165823 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29482
I20260928 05:27:59.166000 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:27:59.166581 8691 external_mini_cluster.cc:1428] Setting key f1f5b2e8c2bed4bfaa327badad46ec72
I20260928 05:27:59.168896 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:27:59.180886 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:59.168322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40753 (local address 127.0.0.1:44149)
0928 05:27:59.169070 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:59.169082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:59.169207 (+ 125us) server_negotiation.cc:407] Connection header received
0928 05:27:59.170946 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:59.170967 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:59.171386 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:59.171730 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:59.172473 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:59.173985 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:59.175853 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:59.176594 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:59.176753 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:59.178008 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:59.178016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:59.178033 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:59.178628 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:59.178724 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:59.178756 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:59.178854 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:59.179193 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 05:27:59.179558 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":326,"threads_started":1}
I20260928 05:27:59.182154 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7: Bootstrap starting.
I20260928 05:27:59.186134 29549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7: Neither blocks nor log segments found. Creating new log.
I20260928 05:27:59.186331 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:27:59.187202 29549 log.cc:824] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:27:59.190656 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7: No bootstrap required, opened a new log
I20260928 05:27:59.200371 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:27:59.193143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58757 (local address 127.0.0.1:44149)
0928 05:27:59.193309 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:27:59.193314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:27:59.193335 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:27:59.193532 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:27:59.193538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:27:59.193643 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:27:59.193726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:27:59.194550 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:59.195669 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:59.197549 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:27:59.198091 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:27:59.198222 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:27:59.199458 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:27:59.199465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:27:59.199470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:27:59.199496 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:27:59.199552 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:27:59.199578 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:27:59.199641 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 05:27:59.199896 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:27:59.200009 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:27:59.201530 29549 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 05:27:59.202095 29549 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:27:59.202214 29549 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6fb5d4ff85e4e71b4f8a6351f5814d7, State: Initialized, Role: FOLLOWER
I20260928 05:27:59.202934 29549 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 05:27:59.203217 29549 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:27:59.203400 29549 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:27:59.203586 29549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:27:59.206069 29549 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 05:27:59.206651 29549 leader_election.cc:302] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6fb5d4ff85e4e71b4f8a6351f5814d7; no voters:
I20260928 05:27:59.207353 29549 leader_election.cc:288] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:27:59.207489 29559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:27:59.209662 29549 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:27:59.210541 29559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [term 1 LEADER]: Becoming Leader. State: Replica: d6fb5d4ff85e4e71b4f8a6351f5814d7, State: Running, Role: LEADER
I20260928 05:27:59.211172 29559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 05:27:59.214044 29561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6fb5d4ff85e4e71b4f8a6351f5814d7. Latest consensus state: current_term: 1 leader_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260928 05:27:59.214169 29560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6fb5d4ff85e4e71b4f8a6351f5814d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260928 05:27:59.214455 29561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:59.214521 29560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:27:59.218148 29570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:27:59.222335 29570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:27:59.230844 29570 catalog_manager.cc:1347] Generated new cluster ID: 39eca4d866264939995a31e744e21e18
I20260928 05:27:59.230963 29570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:27:59.264190 29570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:27:59.266103 29570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:27:59.277755 29570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6fb5d4ff85e4e71b4f8a6351f5814d7: Generated new TSK 0
I20260928 05:27:59.278647 29570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:27:59.287123 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29482
W20260928 05:27:59.288337 29544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40753
W20260928 05:27:59.289125 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:27:59.289279 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29482
2026-09-28T05:27:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:27:59 dist-test-slave-515x krb5kdc[29580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:27:59 dist-test-slave-515x krb5kdc[29580](info): set up 2 sockets
Sep 28 05:27:59 dist-test-slave-515x krb5kdc[29580](info): commencing operation
krb5kdc: starting...
W20260928 05:28:00.305850 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:00 dist-test-slave-515x krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:00.570590 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:00.898399 29593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:00.898725 29593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:00.898857 29593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:00.906514 29593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:00.906652 29593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:00.906733 29593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:00.906821 29593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:00.906879 29593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:00.916880 29593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:00.918190 29593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:00.919487 29593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:00.931185 29598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:00.931206 29601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:00.931241 29599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:00.932101 29593 server_base.cc:1034] running on GCE node
I20260928 05:28:00.933048 29593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:00.935065 29593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:00.936266 29593 hybrid_clock.cc:648] HybridClock initialized: now 1790573280936137 us; error 111 us; skew 500 ppm
I20260928 05:28:00.936766 29593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:00 dist-test-slave-515x krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:00.949811 29593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:00.951390 29593 webserver.cc:460] Webserver started at http://127.0.0.1:38681/ using document root <none> and password file <none>
I20260928 05:28:00.952096 29593 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:00.952214 29593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:00.952507 29593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:00.955577 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca"
format_stamp: "Formatted at 2026-09-28 05:28:00 on dist-test-slave-515x"
server_key: "ce8ed98c6883e207cd3ec639504d969c"
server_key_iv: "b9df96e6f70b152ad460be8d394da561"
server_key_version: "encryptionkey@0"
I20260928 05:28:00.956332 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca"
format_stamp: "Formatted at 2026-09-28 05:28:00 on dist-test-slave-515x"
server_key: "ce8ed98c6883e207cd3ec639504d969c"
server_key_iv: "b9df96e6f70b152ad460be8d394da561"
server_key_version: "encryptionkey@0"
I20260928 05:28:00.961360 29593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:28:00.964674 29608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:00.965976 29593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:00.966127 29593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca"
format_stamp: "Formatted at 2026-09-28 05:28:00 on dist-test-slave-515x"
server_key: "ce8ed98c6883e207cd3ec639504d969c"
server_key_iv: "b9df96e6f70b152ad460be8d394da561"
server_key_version: "encryptionkey@0"
I20260928 05:28:00.966303 29593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:00.971758 29593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:00.976079 29593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:00.976378 29593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:00.992733 29593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39337
I20260928 05:28:00.992779 29659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39337 every 8 connection(s)
I20260928 05:28:00.994153 29593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:00.997888 29660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:01.004304 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29593
I20260928 05:28:01.004488 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:01.005035 8691 external_mini_cluster.cc:1428] Setting key e4a4f3a642a9c82de714ec137a67bcb6
I20260928 05:28:01.008661 29660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca: Bootstrap starting.
I20260928 05:28:01.012794 29660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:01.014161 29660 log.cc:824] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:01.015012 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:01.007155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60359 (local address 127.0.0.1:39337)
0928 05:28:01.008082 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:01.008098 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:01.008558 (+ 460us) server_negotiation.cc:407] Connection header received
0928 05:28:01.010994 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:01.011022 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:01.011496 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:01.011941 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:01.012919 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:01.012929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:01.012944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:01.012962 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:01.013038 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:01.013066 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:01.013281 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:01.013406 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 05:28:01.013594 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":405,"threads_started":1}
I20260928 05:28:01.017299 29660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca: No bootstrap required, opened a new log
I20260928 05:28:01.018676 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:01.026645 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:01.023903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37153 (local address 127.0.0.1:39337)
0928 05:28:01.024080 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:01.024086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:01.024637 (+ 551us) server_negotiation.cc:407] Connection header received
0928 05:28:01.024790 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:01.024797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:01.024849 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:01.024938 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:01.026001 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:01.026005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:01.026008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:01.026024 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:01.026070 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:01.026086 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:01.026202 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:01.026319 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 05:28:01.026369 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:28:01.027266 29660 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260928 05:28:01.027644 29660 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:01.027747 29660 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d020cfe6f34a3e89dfb8e6a72119ca, State: Initialized, Role: FOLLOWER
I20260928 05:28:01.028319 29660 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260928 05:28:01.028544 29660 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:01.028667 29660 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:01.028831 29660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:01.031075 29660 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260928 05:28:01.031555 29660 leader_election.cc:302] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7d020cfe6f34a3e89dfb8e6a72119ca; no voters:
I20260928 05:28:01.032116 29660 leader_election.cc:288] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:01.032270 29670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:01.032878 29670 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [term 1 LEADER]: Becoming Leader. State: Replica: a7d020cfe6f34a3e89dfb8e6a72119ca, State: Running, Role: LEADER
I20260928 05:28:01.033454 29670 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260928 05:28:01.034111 29660 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:01.036638 29672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7d020cfe6f34a3e89dfb8e6a72119ca. Latest consensus state: current_term: 1 leader_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260928 05:28:01.037029 29672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:01.039088 29679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:01.039011 29671 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d020cfe6f34a3e89dfb8e6a72119ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260928 05:28:01.039335 29671 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:01.043376 29679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:01.051409 29679 catalog_manager.cc:1347] Generated new cluster ID: a737498c6d264d659c12266c570a6021
I20260928 05:28:01.051523 29679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:01.073550 29679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:01.074026 29679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:01.100236 29679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7d020cfe6f34a3e89dfb8e6a72119ca: Generated new TSK 0
I20260928 05:28:01.101127 29679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:01.111931 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29593
W20260928 05:28:01.113178 29655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60359
W20260928 05:28:01.113904 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29593: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:01.114046 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29593
2026-09-28T05:28:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1824 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:01 dist-test-slave-515x krb5kdc[29691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:01 dist-test-slave-515x krb5kdc[29691](info): set up 2 sockets
Sep 28 05:28:01 dist-test-slave-515x krb5kdc[29691](info): commencing operation
krb5kdc: starting...
W20260928 05:28:02.137800 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:02 dist-test-slave-515x krb5kdc[29691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:02.388804 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:02.726276 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:02.726567 29704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:02.726701 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:02.733867 29704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:02.733985 29704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:02.734062 29704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:02.734143 29704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:02.734210 29704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:02.744278 29704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:02.745627 29704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:02.746954 29704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:02.758780 29709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:02.758715 29712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:02.758752 29710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:02.763242 29704 server_base.cc:1034] running on GCE node
I20260928 05:28:02.764092 29704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:02.765769 29704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:02.767014 29704 hybrid_clock.cc:648] HybridClock initialized: now 1790573282766969 us; error 33 us; skew 500 ppm
I20260928 05:28:02.767397 29704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:02 dist-test-slave-515x krb5kdc[29691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:02.770654 29704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:02.772338 29704 webserver.cc:460] Webserver started at http://127.0.0.1:33877/ using document root <none> and password file <none>
I20260928 05:28:02.773052 29704 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:02.773172 29704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:02.773465 29704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:02.776497 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "eb2384cea04d49b08661bacc9d16b82d"
format_stamp: "Formatted at 2026-09-28 05:28:02 on dist-test-slave-515x"
server_key: "39cf69f59328af26747d252376e1aee4"
server_key_iv: "d9bcaafb382a8bddcc9f0991f50ceba9"
server_key_version: "encryptionkey@0"
I20260928 05:28:02.777280 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "eb2384cea04d49b08661bacc9d16b82d"
format_stamp: "Formatted at 2026-09-28 05:28:02 on dist-test-slave-515x"
server_key: "39cf69f59328af26747d252376e1aee4"
server_key_iv: "d9bcaafb382a8bddcc9f0991f50ceba9"
server_key_version: "encryptionkey@0"
I20260928 05:28:02.782573 29704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:28:02.786247 29719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:02.787696 29704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:02.787878 29704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "eb2384cea04d49b08661bacc9d16b82d"
format_stamp: "Formatted at 2026-09-28 05:28:02 on dist-test-slave-515x"
server_key: "39cf69f59328af26747d252376e1aee4"
server_key_iv: "d9bcaafb382a8bddcc9f0991f50ceba9"
server_key_version: "encryptionkey@0"
I20260928 05:28:02.788069 29704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:02.802634 29704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:02.807057 29704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:02.807346 29704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:02.825502 29704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33619
I20260928 05:28:02.825533 29770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33619 every 8 connection(s)
I20260928 05:28:02.826948 29704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:02.830705 29771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:02.833994 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29704
I20260928 05:28:02.834192 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:02.834712 8691 external_mini_cluster.cc:1428] Setting key 13e543dfb902850c5e570f095ccb84ce
I20260928 05:28:02.843156 29771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: Bootstrap starting.
I20260928 05:28:02.846966 29771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:02.848274 29771 log.cc:824] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:28:02 dist-test-slave-515x krb5kdc[29691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:28:02.851600 29771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: No bootstrap required, opened a new log
I20260928 05:28:02.858839 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:02.836538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:47651 (local address 127.0.0.1:33619)
0928 05:28:02.837276 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:02.837288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:02.837329 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:28:02.839024 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:02.839048 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:02.839491 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:02.839826 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:02.841639 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:02.843159 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:02.844928 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:02.845775 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:02.845911 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:02.850890 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:02.850898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:02.850914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:02.850962 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:02.854404 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:02.855434 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:02.855440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:02.855450 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:02.855617 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:02.856061 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:02.856064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:02.856066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:02.856633 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:02.856894 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:02.857339 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 05:28:02.857683 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":339,"threads_started":1}
I20260928 05:28:02.860419 29771 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 05:28:02.860785 29771 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:02.860884 29771 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2384cea04d49b08661bacc9d16b82d, State: Initialized, Role: FOLLOWER
I20260928 05:28:02.861461 29771 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 05:28:02.861763 29771 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:02.861888 29771 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:02.862073 29771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:02.862821 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:02.865082 29771 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 05:28:02.865688 29771 leader_election.cc:302] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2384cea04d49b08661bacc9d16b82d; no voters:
I20260928 05:28:02.866338 29771 leader_election.cc:288] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:02.866416 29777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:02.867715 29777 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [term 1 LEADER]: Becoming Leader. State: Replica: eb2384cea04d49b08661bacc9d16b82d, State: Running, Role: LEADER
I20260928 05:28:02.868362 29777 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260928 05:28:02.868692 29771 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:02.873726 29779 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2384cea04d49b08661bacc9d16b82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260928 05:28:02.875243 29781 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2384cea04d49b08661bacc9d16b82d. Latest consensus state: current_term: 1 leader_uuid: "eb2384cea04d49b08661bacc9d16b82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2384cea04d49b08661bacc9d16b82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260928 05:28:02.876508 29781 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:02.876965 29779 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:02.886660 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:02.869452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:33407 (local address 127.0.0.1:33619)
0928 05:28:02.869659 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:02.869668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:02.870191 (+ 523us) server_negotiation.cc:407] Connection header received
0928 05:28:02.870406 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:02.870411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:02.870495 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:02.870607 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:02.872548 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:02.873778 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:02.875019 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:02.875966 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:02.876070 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:02.879374 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:02.879382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:02.879388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:02.879434 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:02.883230 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:02.884237 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:02.884243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:02.884247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:02.884401 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:02.884882 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:02.884886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:02.884889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:02.885341 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:02.885665 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:02.886063 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 05:28:02.886283 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 05:28:02.892931 29795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:28:02.893045 29795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:28:02.893209 29797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:02.895368 29797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:02.904349 29797 catalog_manager.cc:1347] Generated new cluster ID: cbcad9e9b1544751839aca8f3ccc0069
I20260928 05:28:02.904479 29797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:02.929936 29797 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:02.931313 29797 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:02.943203 29797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb2384cea04d49b08661bacc9d16b82d: Generated new TSK 0
I20260928 05:28:02.944146 29797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:02.952916 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29704
I20260928 05:28:02.954559 29766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:03.045984 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29704
W20260928 05:28:03.059885 29689 connection.cc:535] client connection to 127.0.0.1:33619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33619): Connection reset by peer (error 104)
2026-09-28T05:28:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:03 dist-test-slave-515x krb5kdc[29803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:03 dist-test-slave-515x krb5kdc[29803](info): set up 2 sockets
Sep 28 05:28:03 dist-test-slave-515x krb5kdc[29803](info): commencing operation
krb5kdc: starting...
W20260928 05:28:04.085841 8691 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:04 dist-test-slave-515x krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:04.333040 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:04.668390 29816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:04.668707 29816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:04.668843 29816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:04.676257 29816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:04.676383 29816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:04.676441 29816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:04.676522 29816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:04.676581 29816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:04.686844 29816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:04.688222 29816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:04.689558 29816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:04.701898 29821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:04.701956 29824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:04.701978 29822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:04.703090 29816 server_base.cc:1034] running on GCE node
I20260928 05:28:04.703943 29816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:04.705441 29816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:04.706676 29816 hybrid_clock.cc:648] HybridClock initialized: now 1790573284706619 us; error 58 us; skew 500 ppm
I20260928 05:28:04.707016 29816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:04 dist-test-slave-515x krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:04.710280 29816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:04.711814 29816 webserver.cc:460] Webserver started at http://127.0.0.1:35895/ using document root <none> and password file <none>
I20260928 05:28:04.712525 29816 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:04.712634 29816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:04.712924 29816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:04.716073 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "261bf219ebf74531bde4f2cd976bc8f4"
format_stamp: "Formatted at 2026-09-28 05:28:04 on dist-test-slave-515x"
server_key: "9299d665abde0f3068e271fa1ec077a0"
server_key_iv: "bf9e16ce98e90ddfadaf45890dc92fe2"
server_key_version: "encryptionkey@0"
I20260928 05:28:04.716840 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "261bf219ebf74531bde4f2cd976bc8f4"
format_stamp: "Formatted at 2026-09-28 05:28:04 on dist-test-slave-515x"
server_key: "9299d665abde0f3068e271fa1ec077a0"
server_key_iv: "bf9e16ce98e90ddfadaf45890dc92fe2"
server_key_version: "encryptionkey@0"
I20260928 05:28:04.722189 29816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:28:04.725770 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:04.727197 29816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:04.727367 29816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "261bf219ebf74531bde4f2cd976bc8f4"
format_stamp: "Formatted at 2026-09-28 05:28:04 on dist-test-slave-515x"
server_key: "9299d665abde0f3068e271fa1ec077a0"
server_key_iv: "bf9e16ce98e90ddfadaf45890dc92fe2"
server_key_version: "encryptionkey@0"
I20260928 05:28:04.727553 29816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:04.743986 29816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:04.748646 29816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:04.748924 29816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:04.766235 29816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44989
I20260928 05:28:04.766264 29882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44989 every 8 connection(s)
I20260928 05:28:04.767705 29816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:04.771519 29883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:04.777742 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29816
I20260928 05:28:04.777901 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:04.778316 8691 external_mini_cluster.cc:1428] Setting key b8b3fc4f81f4251a42c85bd034ea5d8a
I20260928 05:28:04.782563 29883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4: Bootstrap starting.
I20260928 05:28:04.785892 29883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:04.787104 29883 log.cc:824] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:04.789821 29883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4: No bootstrap required, opened a new log
I20260928 05:28:04.791320 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:04.780032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:53955 (local address 127.0.0.1:44989)
0928 05:28:04.780824 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:04.780836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:04.780875 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:28:04.782516 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:04.782538 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:04.782947 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:04.783284 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:04.783811 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:04.785081 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:04.786909 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:04.787708 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:04.787909 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:04.789060 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:04.789067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:04.789081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:04.789635 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:04.789717 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:04.789746 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:04.789890 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:04.790158 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:28:04.790342 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":361,"threads_started":1}
I20260928 05:28:04.795282 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:04.798038 29883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260928 05:28:04.798448 29883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:04.798554 29883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261bf219ebf74531bde4f2cd976bc8f4, State: Initialized, Role: FOLLOWER
I20260928 05:28:04.799117 29883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260928 05:28:04.799407 29883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:04.799547 29883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:04.799774 29883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:04.802765 29883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260928 05:28:04.803349 29883 leader_election.cc:302] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261bf219ebf74531bde4f2cd976bc8f4; no voters:
I20260928 05:28:04.804030 29883 leader_election.cc:288] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:04.804132 29893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:04.805296 29893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 261bf219ebf74531bde4f2cd976bc8f4, State: Running, Role: LEADER
I20260928 05:28:04.806242 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:04.799447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:47717 (local address 127.0.0.1:44989)
0928 05:28:04.799652 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:04.799658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:04.800033 (+ 375us) server_negotiation.cc:407] Connection header received
0928 05:28:04.800265 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:04.800268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:04.800331 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:04.800414 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:04.800990 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:04.801867 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:04.803578 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:04.804035 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:04.804147 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:04.805433 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:04.805443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:04.805448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:04.805468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:04.805517 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:04.805535 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:04.805672 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:04.805824 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 05:28:04.805881 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:28:04.806473 29883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:04.805997 29893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260928 05:28:04.809335 29895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261bf219ebf74531bde4f2cd976bc8f4. Latest consensus state: current_term: 1 leader_uuid: "261bf219ebf74531bde4f2cd976bc8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260928 05:28:04.809717 29895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:04.810503 29894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261bf219ebf74531bde4f2cd976bc8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261bf219ebf74531bde4f2cd976bc8f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260928 05:28:04.810817 29894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:04.814128 29903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:04.819175 29903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:04.827888 29903 catalog_manager.cc:1347] Generated new cluster ID: 238da66c5f8147158ed2a8ae611780bf
I20260928 05:28:04.828030 29903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:04.838568 29903 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:04.839915 29903 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:04.847810 29903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261bf219ebf74531bde4f2cd976bc8f4: Generated new TSK 0
I20260928 05:28:04.848917 29903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:04.863761 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29816
W20260928 05:28:04.864876 29878 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:53955
W20260928 05:28:04.865628 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:04.865769 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29816
W20260928 05:28:04.879832 29801 connection.cc:535] client connection to 127.0.0.1:44989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44989): Connection reset by peer (error 104)
2026-09-28T05:28:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1819 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:04 dist-test-slave-515x krb5kdc[29914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:04 dist-test-slave-515x krb5kdc[29914](info): set up 2 sockets
Sep 28 05:28:04 dist-test-slave-515x krb5kdc[29914](info): commencing operation
krb5kdc: starting...
W20260928 05:28:05.897848 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:06 dist-test-slave-515x krb5kdc[29914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:06.144984 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:06.467749 29927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:06.468062 29927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:06.468207 29927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:06.475461 29927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:06.475597 29927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:06.475683 29927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:06.475762 29927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:06.475824 29927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:06.485890 29927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:06.487180 29927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:06.488490 29927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:06.500094 29932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:06.500152 29933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:06.500181 29935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:06.500811 29927 server_base.cc:1034] running on GCE node
I20260928 05:28:06.501564 29927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:06.502954 29927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:06.504151 29927 hybrid_clock.cc:648] HybridClock initialized: now 1790573286504099 us; error 48 us; skew 500 ppm
I20260928 05:28:06.504500 29927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:06 dist-test-slave-515x krb5kdc[29914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:06.510979 29927 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:06.512385 29927 webserver.cc:460] Webserver started at http://127.0.0.1:46221/ using document root <none> and password file <none>
I20260928 05:28:06.513015 29927 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:06.513109 29927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:06.513361 29927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:06.516249 29927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "09876762aa894396b94e8504cd1bef37"
format_stamp: "Formatted at 2026-09-28 05:28:06 on dist-test-slave-515x"
server_key: "488b84ca02c2a71e18b6817875d9eba4"
server_key_iv: "185631ce7e384f3fdd441ffb16afb05d"
server_key_version: "encryptionkey@0"
I20260928 05:28:06.516999 29927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "09876762aa894396b94e8504cd1bef37"
format_stamp: "Formatted at 2026-09-28 05:28:06 on dist-test-slave-515x"
server_key: "488b84ca02c2a71e18b6817875d9eba4"
server_key_iv: "185631ce7e384f3fdd441ffb16afb05d"
server_key_version: "encryptionkey@0"
I20260928 05:28:06.522099 29927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:28:06.525570 29942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:06.527022 29927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:06.527222 29927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "09876762aa894396b94e8504cd1bef37"
format_stamp: "Formatted at 2026-09-28 05:28:06 on dist-test-slave-515x"
server_key: "488b84ca02c2a71e18b6817875d9eba4"
server_key_iv: "185631ce7e384f3fdd441ffb16afb05d"
server_key_version: "encryptionkey@0"
I20260928 05:28:06.527401 29927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:06.533032 29927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:06.537262 29927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:06.537523 29927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:06.554464 29927 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34527
I20260928 05:28:06.554497 29993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34527 every 8 connection(s)
I20260928 05:28:06.555796 29927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:06.558768 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 29927
I20260928 05:28:06.558928 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:06.559370 8691 external_mini_cluster.cc:1428] Setting key 62a1aee028e88d34329cab525ff3c18e
I20260928 05:28:06.559568 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:06.567356 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:06.561025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:37277 (local address 127.0.0.1:34527)
0928 05:28:06.561781 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:06.561792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:06.561838 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:28:06.564038 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:06.564061 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:06.564427 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:06.564768 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:06.565845 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:06.565851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:06.565868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:06.565884 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:06.565948 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:06.565975 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:06.566150 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:06.566272 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 05:28:06.566462 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260928 05:28:06.570952 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:06.572609 29994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37: Bootstrap starting.
I20260928 05:28:06.575965 29994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:06.577234 29994 log.cc:824] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:06.577390 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:06.575089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:47525 (local address 127.0.0.1:34527)
0928 05:28:06.575281 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:06.575286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:06.575585 (+ 299us) server_negotiation.cc:407] Connection header received
0928 05:28:06.575749 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:06.575752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:06.575799 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:06.575879 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:06.576765 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:06.576769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:06.576773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:06.576787 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:06.576823 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:06.576837 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:06.576916 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:06.577036 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 05:28:06.577080 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:28:06.579916 29994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37: No bootstrap required, opened a new log
I20260928 05:28:06.586920 29994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } }
I20260928 05:28:06.587270 29994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:06.587359 29994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09876762aa894396b94e8504cd1bef37, State: Initialized, Role: FOLLOWER
I20260928 05:28:06.587895 29994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } }
I20260928 05:28:06.588101 29994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:06.588224 29994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:06.588372 29994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:06.590481 29994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } }
I20260928 05:28:06.590962 29994 leader_election.cc:302] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09876762aa894396b94e8504cd1bef37; no voters:
I20260928 05:28:06.591612 29994 leader_election.cc:288] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:06.591707 30004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:06.592391 30004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [term 1 LEADER]: Becoming Leader. State: Replica: 09876762aa894396b94e8504cd1bef37, State: Running, Role: LEADER
I20260928 05:28:06.592900 30004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } }
I20260928 05:28:06.593688 29994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:06.595530 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09876762aa894396b94e8504cd1bef37. Latest consensus state: current_term: 1 leader_uuid: "09876762aa894396b94e8504cd1bef37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } } }
I20260928 05:28:06.596048 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:06.596021 30005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09876762aa894396b94e8504cd1bef37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09876762aa894396b94e8504cd1bef37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34527 } } }
I20260928 05:28:06.596294 30005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:06.597853 30013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:06.603018 30013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:06.612404 30013 catalog_manager.cc:1347] Generated new cluster ID: 4dccbee874e94726b3ec1bef4be4a642
I20260928 05:28:06.612530 30013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:06.641058 30013 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:06.641496 30013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:06.649078 30013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09876762aa894396b94e8504cd1bef37: Generated new TSK 0
I20260928 05:28:06.649995 30013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:06.662374 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29927
W20260928 05:28:06.663533 29989 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:37277
W20260928 05:28:06.664201 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 29927: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:06.664333 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 29927
2026-09-28T05:28:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:06 dist-test-slave-515x krb5kdc[30025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:06 dist-test-slave-515x krb5kdc[30025](info): set up 2 sockets
Sep 28 05:28:06 dist-test-slave-515x krb5kdc[30025](info): commencing operation
krb5kdc: starting...
W20260928 05:28:07.669845 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:07 dist-test-slave-515x krb5kdc[30025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:07.917294 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:08.236557 30038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:08.236852 30038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:08.236970 30038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:08.244323 30038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:08.244441 30038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:08.244513 30038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:08.244593 30038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:08.244653 30038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:08.254822 30038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:08.256163 30038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:08.257488 30038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:08.269541 30044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:08.269629 30046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:08.270153 30038 server_base.cc:1034] running on GCE node
W20260928 05:28:08.270483 30043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:08.271387 30038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:08.272974 30038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:08.274295 30038 hybrid_clock.cc:648] HybridClock initialized: now 1790573288274218 us; error 65 us; skew 500 ppm
I20260928 05:28:08.274684 30038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:08 dist-test-slave-515x krb5kdc[30025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:08.277930 30038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:08.279520 30038 webserver.cc:460] Webserver started at http://127.0.0.1:38757/ using document root <none> and password file <none>
I20260928 05:28:08.280265 30038 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:08.280383 30038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:08.280675 30038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:08.283865 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "4a489391879c4023b453bc9999c85a35"
format_stamp: "Formatted at 2026-09-28 05:28:08 on dist-test-slave-515x"
server_key: "fc5829ebcaebd92d9df564da2d3f2b86"
server_key_iv: "47687752973dc6655da3e5cf45c2ceb7"
server_key_version: "encryptionkey@0"
I20260928 05:28:08.284641 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "4a489391879c4023b453bc9999c85a35"
format_stamp: "Formatted at 2026-09-28 05:28:08 on dist-test-slave-515x"
server_key: "fc5829ebcaebd92d9df564da2d3f2b86"
server_key_iv: "47687752973dc6655da3e5cf45c2ceb7"
server_key_version: "encryptionkey@0"
I20260928 05:28:08.290196 30038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:28:08.293848 30053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:08.295255 30038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:08.295415 30038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "4a489391879c4023b453bc9999c85a35"
format_stamp: "Formatted at 2026-09-28 05:28:08 on dist-test-slave-515x"
server_key: "fc5829ebcaebd92d9df564da2d3f2b86"
server_key_iv: "47687752973dc6655da3e5cf45c2ceb7"
server_key_version: "encryptionkey@0"
I20260928 05:28:08.295598 30038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:08.316924 30038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:08.321014 30038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:08.321282 30038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:08.337772 30038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45055
I20260928 05:28:08.337803 30104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45055 every 8 connection(s)
I20260928 05:28:08.339067 30038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:08.340822 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30038
I20260928 05:28:08.341001 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:08.341581 8691 external_mini_cluster.cc:1428] Setting key d67203c1e0c1f307b7df4ef0071501ac
I20260928 05:28:08.343344 30105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:28:08 dist-test-slave-515x krb5kdc[30025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:28:08.356395 30105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: Bootstrap starting.
I20260928 05:28:08.360244 30105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:08.361440 30105 log.cc:824] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:08.364758 30105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: No bootstrap required, opened a new log
I20260928 05:28:08.365265 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:08.343360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:52003 (local address 127.0.0.1:45055)
0928 05:28:08.344119 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:08.344134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:08.344193 (+ 59us) server_negotiation.cc:407] Connection header received
0928 05:28:08.346345 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:08.346372 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:08.346887 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:08.347311 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:08.348592 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:08.350605 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:08.351758 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:08.352755 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:08.352930 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:08.357082 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:08.357093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:08.357111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:08.357179 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:08.360810 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:08.361595 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:08.361628 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:08.361636 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:08.361848 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:08.362350 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:08.362354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:08.362356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:08.362955 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:08.363317 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:08.363604 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 05:28:08.363929 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":293,"threads_started":1}
I20260928 05:28:08.369429 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:08.374648 30105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260928 05:28:08.375079 30105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:08.375186 30105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a489391879c4023b453bc9999c85a35, State: Initialized, Role: FOLLOWER
I20260928 05:28:08.375797 30105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260928 05:28:08.376014 30105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:08.376140 30105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:08.376297 30105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:08.378566 30105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260928 05:28:08.379042 30105 leader_election.cc:302] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a489391879c4023b453bc9999c85a35; no voters:
I20260928 05:28:08.379679 30105 leader_election.cc:288] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:08.379755 30115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:08.382051 30105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:08.385862 30115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [term 1 LEADER]: Becoming Leader. State: Replica: 4a489391879c4023b453bc9999c85a35, State: Running, Role: LEADER
I20260928 05:28:08.386557 30115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260928 05:28:08.388173 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:08.374592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:36825 (local address 127.0.0.1:45055)
0928 05:28:08.374777 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:08.374783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:08.375167 (+ 384us) server_negotiation.cc:407] Connection header received
0928 05:28:08.375363 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:08.375367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:08.375429 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:08.375510 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:08.376996 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:08.377858 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:08.379127 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:08.379662 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:08.379784 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:08.382364 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:08.382369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:08.382372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:08.382403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:08.385267 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:08.386154 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:08.386158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:08.386160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:08.386300 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:08.386714 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:08.386719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:08.386723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:08.387038 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:08.387313 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:08.387636 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 05:28:08.387801 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:28:08.394135 30116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a489391879c4023b453bc9999c85a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } } }
I20260928 05:28:08.394517 30116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:08.395125 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a489391879c4023b453bc9999c85a35. Latest consensus state: current_term: 1 leader_uuid: "4a489391879c4023b453bc9999c85a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a489391879c4023b453bc9999c85a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } } }
I20260928 05:28:08.395376 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35 [sys.catalog]: This master's current role is: LEADER
W20260928 05:28:08.402074 30130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:28:08.402179 30130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:28:08.402283 30129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:08.403652 30129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:08.411994 30129 catalog_manager.cc:1347] Generated new cluster ID: 037397b1f6ed4752a98d528a50359c8c
I20260928 05:28:08.412103 30129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:08.446231 30129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:08.447496 30129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:08.460199 30129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a489391879c4023b453bc9999c85a35: Generated new TSK 0
I20260928 05:28:08.461114 30129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:08.469936 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30038
I20260928 05:28:08.471113 30100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:08.563252 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30038
W20260928 05:28:08.577333 30023 connection.cc:535] client connection to 127.0.0.1:45055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45055): Connection reset by peer (error 104)
2026-09-28T05:28:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:08 dist-test-slave-515x krb5kdc[30137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:08 dist-test-slave-515x krb5kdc[30137](info): set up 2 sockets
Sep 28 05:28:08 dist-test-slave-515x krb5kdc[30137](info): commencing operation
krb5kdc: starting...
W20260928 05:28:09.582013 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:09 dist-test-slave-515x krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:09.818955 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:10.135076 30150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:10.135385 30150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:10.135526 30150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:10.142637 30150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:10.142757 30150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:10.142835 30150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:10.142915 30150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:10.142973 30150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:10.152621 30150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:10.153914 30150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:10.155208 30150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:10.166507 30155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:10.166450 30156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:10.166618 30158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:10.167192 30150 server_base.cc:1034] running on GCE node
I20260928 05:28:10.167954 30150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:10.169339 30150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:10.170544 30150 hybrid_clock.cc:648] HybridClock initialized: now 1790573290170491 us; error 42 us; skew 500 ppm
I20260928 05:28:10.170868 30150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:10 dist-test-slave-515x krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:10.173895 30150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:10.175416 30150 webserver.cc:460] Webserver started at http://127.0.0.1:42729/ using document root <none> and password file <none>
I20260928 05:28:10.176090 30150 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:10.176205 30150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:10.176482 30150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:10.179499 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "b77e3481daa045218d939ba9d50d7592"
format_stamp: "Formatted at 2026-09-28 05:28:10 on dist-test-slave-515x"
server_key: "e9ea652071ec28f64bd917e6be207730"
server_key_iv: "8407a13f76899c0a50b358630f2f793f"
server_key_version: "encryptionkey@0"
I20260928 05:28:10.180318 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "b77e3481daa045218d939ba9d50d7592"
format_stamp: "Formatted at 2026-09-28 05:28:10 on dist-test-slave-515x"
server_key: "e9ea652071ec28f64bd917e6be207730"
server_key_iv: "8407a13f76899c0a50b358630f2f793f"
server_key_version: "encryptionkey@0"
I20260928 05:28:10.185420 30150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:28:10.188803 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:10.190178 30150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:10.190338 30150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "b77e3481daa045218d939ba9d50d7592"
format_stamp: "Formatted at 2026-09-28 05:28:10 on dist-test-slave-515x"
server_key: "e9ea652071ec28f64bd917e6be207730"
server_key_iv: "8407a13f76899c0a50b358630f2f793f"
server_key_version: "encryptionkey@0"
I20260928 05:28:10.190537 30150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:10.207825 30150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:10.212035 30150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:10.212261 30150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:10.230484 30150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38725
I20260928 05:28:10.230511 30216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38725 every 8 connection(s)
I20260928 05:28:10.231840 30150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:10.235554 30217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:10.241690 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30150
I20260928 05:28:10.241834 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:10.242269 8691 external_mini_cluster.cc:1428] Setting key c3c04f0a5bc602dc61f33dcc940a5d1a
I20260928 05:28:10.245973 30217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592: Bootstrap starting.
I20260928 05:28:10.249203 30217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:10.250504 30217 log.cc:824] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:10.253743 30217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592: No bootstrap required, opened a new log
I20260928 05:28:10.255707 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:10.243971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:50181 (local address 127.0.0.1:38725)
0928 05:28:10.244681 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:10.244694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:10.244741 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:28:10.246441 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:10.246463 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:10.246863 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:10.247171 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:10.247772 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:10.249019 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:10.250882 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:10.251907 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:10.252031 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:10.253265 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:10.253273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:10.253288 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:10.253887 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:10.253972 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:10.253999 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:10.254122 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:10.254461 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 05:28:10.254691 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":321,"threads_started":1}
I20260928 05:28:10.259557 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:10.262625 30217 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260928 05:28:10.263096 30217 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:10.263222 30217 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77e3481daa045218d939ba9d50d7592, State: Initialized, Role: FOLLOWER
I20260928 05:28:10.263895 30217 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260928 05:28:10.264161 30217 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:10.264312 30217 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:10.264482 30217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:10.267644 30217 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260928 05:28:10.268123 30217 leader_election.cc:302] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77e3481daa045218d939ba9d50d7592; no voters:
I20260928 05:28:10.268695 30217 leader_election.cc:288] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:10.268848 30227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:10.269433 30227 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [term 1 LEADER]: Becoming Leader. State: Replica: b77e3481daa045218d939ba9d50d7592, State: Running, Role: LEADER
I20260928 05:28:10.270073 30227 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260928 05:28:10.272058 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:10.264779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:36095 (local address 127.0.0.1:38725)
0928 05:28:10.264957 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:10.264963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:10.265259 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:28:10.265409 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:10.265418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:10.265545 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:10.265754 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:10.266162 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:10.267294 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:10.268748 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:10.269474 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:10.269634 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:10.271236 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:10.271243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:10.271248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:10.271269 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:10.271342 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:10.271363 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:10.271480 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:10.271653 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 05:28:10.271714 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:28:10.276856 30217 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:10.280834 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77e3481daa045218d939ba9d50d7592. Latest consensus state: current_term: 1 leader_uuid: "b77e3481daa045218d939ba9d50d7592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260928 05:28:10.280735 30229 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77e3481daa045218d939ba9d50d7592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77e3481daa045218d939ba9d50d7592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260928 05:28:10.281250 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:10.281301 30229 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:10.282202 30242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:10.286424 30242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:10.294478 30242 catalog_manager.cc:1347] Generated new cluster ID: 744f65114ed04f7189763ddc6c4918fa
I20260928 05:28:10.294718 30242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:10.315210 30242 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:10.317058 30242 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:10.324120 30242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77e3481daa045218d939ba9d50d7592: Generated new TSK 0
I20260928 05:28:10.325167 30242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:10.334064 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30150
W20260928 05:28:10.335219 30212 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:50181
W20260928 05:28:10.335973 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:10.336102 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30150
W20260928 05:28:10.350076 30135 connection.cc:535] client connection to 127.0.0.1:38725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38725): Connection reset by peer (error 104)
2026-09-28T05:28:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1772 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:10 dist-test-slave-515x krb5kdc[30249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:10 dist-test-slave-515x krb5kdc[30249](info): set up 2 sockets
Sep 28 05:28:10 dist-test-slave-515x krb5kdc[30249](info): commencing operation
krb5kdc: starting...
W20260928 05:28:11.349754 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:11 dist-test-slave-515x krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:28:11.591001 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:11.903771 30262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:11.904054 30262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:11.904214 30262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:11.911484 30262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:11.911614 30262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:11.911695 30262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:11.911775 30262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:11.911834 30262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:11.921231 30262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:11.922408 30262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:11.923616 30262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:11.934754 30267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:11.934701 30268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:11.934710 30270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:11.935570 30262 server_base.cc:1034] running on GCE node
I20260928 05:28:11.936353 30262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:11.937821 30262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:11.939045 30262 hybrid_clock.cc:648] HybridClock initialized: now 1790573291938999 us; error 38 us; skew 500 ppm
I20260928 05:28:11.939450 30262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:11 dist-test-slave-515x krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:11.946491 30262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:28:11.947996 30262 webserver.cc:460] Webserver started at http://127.0.0.1:43677/ using document root <none> and password file <none>
I20260928 05:28:11.948662 30262 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:11.948779 30262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:11.949059 30262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:11.952057 30262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "941a3be6a5f3491f87cedee09081be52"
format_stamp: "Formatted at 2026-09-28 05:28:11 on dist-test-slave-515x"
server_key: "ef6277c8aa580f8bc1438542aaea967f"
server_key_iv: "c2f2c21ee1bad2f9493a2adca69f2016"
server_key_version: "encryptionkey@0"
I20260928 05:28:11.952808 30262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "941a3be6a5f3491f87cedee09081be52"
format_stamp: "Formatted at 2026-09-28 05:28:11 on dist-test-slave-515x"
server_key: "ef6277c8aa580f8bc1438542aaea967f"
server_key_iv: "c2f2c21ee1bad2f9493a2adca69f2016"
server_key_version: "encryptionkey@0"
I20260928 05:28:11.957895 30262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:28:11.961422 30277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:11.962774 30262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:11.962932 30262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "941a3be6a5f3491f87cedee09081be52"
format_stamp: "Formatted at 2026-09-28 05:28:11 on dist-test-slave-515x"
server_key: "ef6277c8aa580f8bc1438542aaea967f"
server_key_iv: "c2f2c21ee1bad2f9493a2adca69f2016"
server_key_version: "encryptionkey@0"
I20260928 05:28:11.963110 30262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:11.968427 30262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:11.972106 30262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:11.972362 30262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:11.988706 30262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41663
I20260928 05:28:11.988749 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41663 every 8 connection(s)
I20260928 05:28:11.990000 30262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:11.992396 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30262
I20260928 05:28:11.992548 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:11.992962 8691 external_mini_cluster.cc:1428] Setting key c5485de2807225a1eb69af6880c0bc55
I20260928 05:28:11.993647 30329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:12.000726 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:11.994705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:36017 (local address 127.0.0.1:41663)
0928 05:28:11.995466 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:11.995479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:11.995527 (+ 48us) server_negotiation.cc:407] Connection header received
0928 05:28:11.997035 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:11.997061 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:11.997515 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:11.997968 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:11.998888 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:11.998896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:11.998912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:11.998928 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:11.999016 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:11.999039 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:11.999260 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:11.999385 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 05:28:11.999587 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":300,"threads_started":1}
I20260928 05:28:12.003963 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:12.006682 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52: Bootstrap starting.
I20260928 05:28:12.009940 30329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:12.010160 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:12.007907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:54261 (local address 127.0.0.1:41663)
0928 05:28:12.008090 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:12.008096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:12.008343 (+ 247us) server_negotiation.cc:407] Connection header received
0928 05:28:12.008463 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:12.008472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:12.008534 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:12.008635 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:12.009406 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:12.009411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:12.009415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:12.009433 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:12.009476 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:12.009493 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:12.009624 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:12.009791 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 05:28:12.009851 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:28:12.011076 30329 log.cc:824] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:12.013571 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52: No bootstrap required, opened a new log
I20260928 05:28:12.020494 30329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 05:28:12.020852 30329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:12.020946 30329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941a3be6a5f3491f87cedee09081be52, State: Initialized, Role: FOLLOWER
I20260928 05:28:12.021458 30329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 05:28:12.021715 30329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:12.021839 30329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:12.021986 30329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:12.024050 30329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 05:28:12.024515 30329 leader_election.cc:302] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941a3be6a5f3491f87cedee09081be52; no voters:
I20260928 05:28:12.025053 30329 leader_election.cc:288] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:12.025185 30339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:12.025784 30339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [term 1 LEADER]: Becoming Leader. State: Replica: 941a3be6a5f3491f87cedee09081be52, State: Running, Role: LEADER
I20260928 05:28:12.026288 30339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 05:28:12.026947 30329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:12.028865 30341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941a3be6a5f3491f87cedee09081be52. Latest consensus state: current_term: 1 leader_uuid: "941a3be6a5f3491f87cedee09081be52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 05:28:12.029377 30341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:12.029047 30340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941a3be6a5f3491f87cedee09081be52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941a3be6a5f3491f87cedee09081be52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 05:28:12.029836 30340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:12.031143 30348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:12.035647 30348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:12.043430 30348 catalog_manager.cc:1347] Generated new cluster ID: 65d5e07a5a154448b9f40f1b06e625e2
I20260928 05:28:12.043537 30348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:12.070778 30348 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:12.071306 30348 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:12.085516 30348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941a3be6a5f3491f87cedee09081be52: Generated new TSK 0
I20260928 05:28:12.086620 30348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:12.092141 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30262
W20260928 05:28:12.093170 30324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.58:36017
W20260928 05:28:12.093797 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30262: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:12.093917 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30262
2026-09-28T05:28:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1757 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:12 dist-test-slave-515x krb5kdc[30360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:12 dist-test-slave-515x krb5kdc[30360](info): set up 2 sockets
Sep 28 05:28:12 dist-test-slave-515x krb5kdc[30360](info): commencing operation
krb5kdc: starting...
W20260928 05:28:13.097784 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:13 dist-test-slave-515x krb5kdc[30360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:13.337201 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44225
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:44579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:13.657099 30373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:13.657415 30373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:13.657553 30373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:13.664690 30373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:13.664793 30373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:13.664850 30373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:13.664908 30373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:13.664949 30373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:13.674610 30373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:44579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:44225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:13.675832 30373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:13.677067 30373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:13.687924 30378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:13.687858 30381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:13.687898 30379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:13.688812 30373 server_base.cc:1034] running on GCE node
I20260928 05:28:13.689486 30373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:13.690824 30373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:13.692013 30373 hybrid_clock.cc:648] HybridClock initialized: now 1790573293691980 us; error 26 us; skew 500 ppm
I20260928 05:28:13.692297 30373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:13 dist-test-slave-515x krb5kdc[30360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573293, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:13.695192 30373 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:13.696475 30373 webserver.cc:460] Webserver started at http://127.8.124.254:45387/ using document root <none> and password file <none>
I20260928 05:28:13.697073 30373 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:13.697182 30373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:13.697461 30373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:13.700493 30373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "c2816cf7cbb84ca39602bdfc71c54271"
format_stamp: "Formatted at 2026-09-28 05:28:13 on dist-test-slave-515x"
server_key: "80a6a21c02018ad1eb31efb738beb0a2"
server_key_iv: "0997c85dd020622752a97bd7b9ca5462"
server_key_version: "encryptionkey@0"
I20260928 05:28:13.701278 30373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "c2816cf7cbb84ca39602bdfc71c54271"
format_stamp: "Formatted at 2026-09-28 05:28:13 on dist-test-slave-515x"
server_key: "80a6a21c02018ad1eb31efb738beb0a2"
server_key_iv: "0997c85dd020622752a97bd7b9ca5462"
server_key_version: "encryptionkey@0"
I20260928 05:28:13.706097 30373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260928 05:28:13.709388 30388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:13.710654 30373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:13.710804 30373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "c2816cf7cbb84ca39602bdfc71c54271"
format_stamp: "Formatted at 2026-09-28 05:28:13 on dist-test-slave-515x"
server_key: "80a6a21c02018ad1eb31efb738beb0a2"
server_key_iv: "0997c85dd020622752a97bd7b9ca5462"
server_key_version: "encryptionkey@0"
I20260928 05:28:13.710969 30373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:13.727680 30373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:13.731330 30373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:13.731562 30373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:13.747963 30373 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:44225
I20260928 05:28:13.747992 30439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:44225 every 8 connection(s)
I20260928 05:28:13.749200 30373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:13.752655 30440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:13.759292 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30373
I20260928 05:28:13.759490 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:13.760015 8691 external_mini_cluster.cc:1428] Setting key aa8c8836282ba0fbc11bc59d12949a88
I20260928 05:28:13.763182 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271: Bootstrap starting.
I20260928 05:28:13.766690 30440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:13.767684 30440 log.cc:824] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:13.770375 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271: No bootstrap required, opened a new log
Sep 28 05:28:13 dist-test-slave-515x krb5kdc[30360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:13.778308 30440 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } }
I20260928 05:28:13.778620 30440 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:13.778716 30440 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2816cf7cbb84ca39602bdfc71c54271, State: Initialized, Role: FOLLOWER
I20260928 05:28:13.779276 30440 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } }
I20260928 05:28:13.779556 30440 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:13.779695 30440 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:13.779850 30440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:13.782102 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:13.761781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43631 (local address 127.8.124.254:44225)
0928 05:28:13.762485 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:13.762497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:13.762536 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:28:13.764220 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:13.764240 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:13.764619 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:13.764977 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:13.766655 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:13.768025 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:13.769654 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:13.770352 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:13.770633 (+ 281us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:13.775093 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:13.775102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:13.775116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:13.775163 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:13.778073 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:13.778790 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:13.778796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:13.778801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:13.778979 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:13.779459 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:13.779463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:13.779465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:13.780038 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:13.780327 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:13.780772 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 05:28:13.781082 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
I20260928 05:28:13.782397 30440 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } }
I20260928 05:28:13.782931 30440 leader_election.cc:302] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2816cf7cbb84ca39602bdfc71c54271; no voters:
I20260928 05:28:13.783500 30440 leader_election.cc:288] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:13.783624 30445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:13.784886 30445 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [term 1 LEADER]: Becoming Leader. State: Replica: c2816cf7cbb84ca39602bdfc71c54271, State: Running, Role: LEADER
I20260928 05:28:13.785471 30440 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:13.785423 30445 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } }
I20260928 05:28:13.786651 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:13.791584 30447 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2816cf7cbb84ca39602bdfc71c54271. Latest consensus state: current_term: 1 leader_uuid: "c2816cf7cbb84ca39602bdfc71c54271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } } }
I20260928 05:28:13.791944 30447 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:13.794162 30446 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2816cf7cbb84ca39602bdfc71c54271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2816cf7cbb84ca39602bdfc71c54271" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44225 } } }
I20260928 05:28:13.794436 30446 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:13.799638 30456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:13.805043 30456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:13.824777 30456 catalog_manager.cc:1347] Generated new cluster ID: 7cd4f47212364fa58d7d5666334d7d50
I20260928 05:28:13.824888 30456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:13.834867 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:13.808566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37623 (local address 127.8.124.254:44225)
0928 05:28:13.808771 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:13.808779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:13.822213 (+ 13434us) server_negotiation.cc:407] Connection header received
0928 05:28:13.822440 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:13.822447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:13.822541 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:13.822660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:13.824469 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:13.825326 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:13.826603 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:13.827016 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:13.827121 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:13.829426 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:13.829431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:13.829437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:13.829467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:13.832156 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:13.832847 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:13.832851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:13.832854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:13.832976 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:13.833397 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:13.833401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:13.833404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:13.833746 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:13.834003 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:13.834355 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 05:28:13.834518 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:28:13.839996 30456 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:13.841547 30456 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:13.847839 30456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2816cf7cbb84ca39602bdfc71c54271: Generated new TSK 0
I20260928 05:28:13.848580 30456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:13.857281 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30373
I20260928 05:28:13.858412 30435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:13.946585 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30373
W20260928 05:28:13.960868 30358 connection.cc:535] client connection to 127.8.124.254:44225 recv error: Network error: failed to read from TLS socket (remote: 127.8.124.254:44225): Connection reset by peer (error 104)
2026-09-28T05:28:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:14 dist-test-slave-515x krb5kdc[30472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:14 dist-test-slave-515x krb5kdc[30472](info): set up 2 sockets
Sep 28 05:28:14 dist-test-slave-515x krb5kdc[30472](info): commencing operation
krb5kdc: starting...
W20260928 05:28:14.949855 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:15 dist-test-slave-515x krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:15.185429 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44887
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:44887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:15.500396 30485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:15.500680 30485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:15.500805 30485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:15.507823 30485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:15.507944 30485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:15.508021 30485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:15.508106 30485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:15.508172 30485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:15.517652 30485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:44887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:15.518839 30485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:15.520082 30485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:15.530907 30493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:15.530982 30491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:15.530987 30490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:15.531596 30485 server_base.cc:1034] running on GCE node
I20260928 05:28:15.532280 30485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:15.533591 30485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:15.534833 30485 hybrid_clock.cc:648] HybridClock initialized: now 1790573295534799 us; error 27 us; skew 500 ppm
I20260928 05:28:15.535202 30485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:15 dist-test-slave-515x krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573295, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:15.538318 30485 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:15.539868 30485 webserver.cc:460] Webserver started at http://127.8.124.254:44611/ using document root <none> and password file <none>
I20260928 05:28:15.540500 30485 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:15.540612 30485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:15.540896 30485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:15.543798 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "74d42721c2594a2eab7d4401a83a5de4"
format_stamp: "Formatted at 2026-09-28 05:28:15 on dist-test-slave-515x"
server_key: "5c7f20bef0805f53f371bdb24cbfd067"
server_key_iv: "cb870ea8ade43f51e715002106f1b71c"
server_key_version: "encryptionkey@0"
I20260928 05:28:15.544513 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "74d42721c2594a2eab7d4401a83a5de4"
format_stamp: "Formatted at 2026-09-28 05:28:15 on dist-test-slave-515x"
server_key: "5c7f20bef0805f53f371bdb24cbfd067"
server_key_iv: "cb870ea8ade43f51e715002106f1b71c"
server_key_version: "encryptionkey@0"
I20260928 05:28:15.549481 30485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 05:28:15.552935 30500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:15.554282 30485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:15.554430 30485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "74d42721c2594a2eab7d4401a83a5de4"
format_stamp: "Formatted at 2026-09-28 05:28:15 on dist-test-slave-515x"
server_key: "5c7f20bef0805f53f371bdb24cbfd067"
server_key_iv: "cb870ea8ade43f51e715002106f1b71c"
server_key_version: "encryptionkey@0"
I20260928 05:28:15.554600 30485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:15.583395 30485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:15.587558 30485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:15.587814 30485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:15.604278 30485 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:44887
I20260928 05:28:15.604287 30551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:44887 every 8 connection(s)
I20260928 05:28:15.605523 30485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:15.609117 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30485
I20260928 05:28:15.609164 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:15.609261 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:15.609779 8691 external_mini_cluster.cc:1428] Setting key 76550a94daaa7579d95b97986695fa4d
I20260928 05:28:15.623476 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4: Bootstrap starting.
I20260928 05:28:15.623733 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:15.611599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46451 (local address 127.8.124.254:44887)
0928 05:28:15.612341 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:15.612359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:15.612407 (+ 48us) server_negotiation.cc:407] Connection header received
0928 05:28:15.614087 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:15.614118 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:15.614615 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:15.615064 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:15.615572 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:15.617495 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:15.618895 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:15.619902 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:15.620077 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:15.621123 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:15.621133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:15.621151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:15.621855 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:15.621940 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:15.621975 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:15.622126 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:15.622347 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 05:28:15.622507 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":313,"threads_started":1}
I20260928 05:28:15.626602 30552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:15.627167 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:15.627693 30552 log.cc:824] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:15.630781 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4: No bootstrap required, opened a new log
I20260928 05:28:15.638216 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:15.631998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33967 (local address 127.8.124.254:44887)
0928 05:28:15.632151 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:15.632156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:15.632474 (+ 318us) server_negotiation.cc:407] Connection header received
0928 05:28:15.632614 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:15.632619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:15.632697 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:15.632794 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:15.633191 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:15.634332 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:15.635588 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:15.636289 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:15.636419 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:15.637336 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:15.637343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:15.637347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:15.637367 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:15.637428 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:15.637447 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:15.637561 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:15.637830 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 05:28:15.637883 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:28:15.639948 30552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } }
I20260928 05:28:15.640298 30552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:15.640372 30552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d42721c2594a2eab7d4401a83a5de4, State: Initialized, Role: FOLLOWER
I20260928 05:28:15.640851 30552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } }
I20260928 05:28:15.641063 30552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:15.641170 30552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:15.641291 30552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:15.643328 30552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } }
I20260928 05:28:15.643750 30552 leader_election.cc:302] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74d42721c2594a2eab7d4401a83a5de4; no voters:
I20260928 05:28:15.644259 30552 leader_election.cc:288] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:15.644374 30562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:15.644977 30562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [term 1 LEADER]: Becoming Leader. State: Replica: 74d42721c2594a2eab7d4401a83a5de4, State: Running, Role: LEADER
I20260928 05:28:15.645547 30562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } }
I20260928 05:28:15.646420 30552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:15.648170 30563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74d42721c2594a2eab7d4401a83a5de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } } }
I20260928 05:28:15.648538 30563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:15.648196 30564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74d42721c2594a2eab7d4401a83a5de4. Latest consensus state: current_term: 1 leader_uuid: "74d42721c2594a2eab7d4401a83a5de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d42721c2594a2eab7d4401a83a5de4" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 44887 } } }
I20260928 05:28:15.649430 30564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:15.651521 30569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:15.657169 30569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:15.670799 30569 catalog_manager.cc:1347] Generated new cluster ID: f641bec048924fee95f0b850625b7687
I20260928 05:28:15.670917 30569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:15.678753 30569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:15.680603 30569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:15.688982 30569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74d42721c2594a2eab7d4401a83a5de4: Generated new TSK 0
I20260928 05:28:15.689828 30569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:15.703320 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30485
W20260928 05:28:15.704495 30547 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46451
W20260928 05:28:15.705279 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:15.705422 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30485
W20260928 05:28:15.719125 30470 connection.cc:535] client connection to 127.8.124.254:44887 recv error: Network error: failed to read from TLS socket (remote: 127.8.124.254:44887): Connection reset by peer (error 104)
2026-09-28T05:28:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:15 dist-test-slave-515x krb5kdc[30583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:15 dist-test-slave-515x krb5kdc[30583](info): set up 2 sockets
Sep 28 05:28:15 dist-test-slave-515x krb5kdc[30583](info): commencing operation
krb5kdc: starting...
W20260928 05:28:16.713847 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:16 dist-test-slave-515x krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:16.940608 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41379
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:17.255441 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:17.255735 30596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:17.255877 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:17.263098 30596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:17.263223 30596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:17.263303 30596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:17.263386 30596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:17.263442 30596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:17.273402 30596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:41379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:17.274699 30596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:17.276012 30596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:17.287328 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:17.287281 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:17.287352 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:17.288043 30596 server_base.cc:1034] running on GCE node
I20260928 05:28:17.288779 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:17.290189 30596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:17.291379 30596 hybrid_clock.cc:648] HybridClock initialized: now 1790573297291340 us; error 38 us; skew 500 ppm
I20260928 05:28:17.291697 30596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:17 dist-test-slave-515x krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573297, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:17.296950 30596 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:17.298417 30596 webserver.cc:460] Webserver started at http://127.8.124.254:39489/ using document root <none> and password file <none>
I20260928 05:28:17.299041 30596 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:17.299154 30596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:17.299424 30596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:17.302311 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "483ae408a4b74a44976ae2018f331ba0"
format_stamp: "Formatted at 2026-09-28 05:28:17 on dist-test-slave-515x"
server_key: "beaef5bfde71eaa298b96355e2b3fcf9"
server_key_iv: "6fb4101ea08da41212cc6512f4f20bd4"
server_key_version: "encryptionkey@0"
I20260928 05:28:17.303022 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "483ae408a4b74a44976ae2018f331ba0"
format_stamp: "Formatted at 2026-09-28 05:28:17 on dist-test-slave-515x"
server_key: "beaef5bfde71eaa298b96355e2b3fcf9"
server_key_iv: "6fb4101ea08da41212cc6512f4f20bd4"
server_key_version: "encryptionkey@0"
I20260928 05:28:17.307822 30596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:28:17.311169 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:17.312463 30596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:28:17.312616 30596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "483ae408a4b74a44976ae2018f331ba0"
format_stamp: "Formatted at 2026-09-28 05:28:17 on dist-test-slave-515x"
server_key: "beaef5bfde71eaa298b96355e2b3fcf9"
server_key_iv: "6fb4101ea08da41212cc6512f4f20bd4"
server_key_version: "encryptionkey@0"
I20260928 05:28:17.312798 30596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:17.318320 30596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:17.322427 30596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:17.322692 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:17.339540 30596 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:41379
I20260928 05:28:17.339567 30662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:41379 every 8 connection(s)
I20260928 05:28:17.340888 30596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:17.342499 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30596
I20260928 05:28:17.342684 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:17.343243 8691 external_mini_cluster.cc:1428] Setting key 9484df95f45bc088b293497fc899d6d3
I20260928 05:28:17.345144 30663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:17.351099 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:17.345063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33755 (local address 127.8.124.254:41379)
0928 05:28:17.345950 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:17.345965 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:17.346282 (+ 317us) server_negotiation.cc:407] Connection header received
0928 05:28:17.347900 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:17.347921 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:17.348243 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:17.348561 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:17.349620 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:17.349628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:17.349645 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:17.349659 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:17.349734 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:17.349755 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:17.349926 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:17.350018 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:28:17.350207 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":352,"threads_started":1}
I20260928 05:28:17.354568 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:17.357340 30663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0: Bootstrap starting.
I20260928 05:28:17.360942 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:17.358544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35689 (local address 127.8.124.254:41379)
0928 05:28:17.358742 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:17.358748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:17.359217 (+ 469us) server_negotiation.cc:407] Connection header received
0928 05:28:17.359383 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:17.359387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:17.359436 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:17.359509 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:17.360380 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:17.360384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:17.360387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:17.360400 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:17.360436 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:17.360451 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:17.360537 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:17.360651 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 05:28:17.360691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:28:17.361030 30663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:17.362111 30663 log.cc:824] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:17.364609 30663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0: No bootstrap required, opened a new log
I20260928 05:28:17.371397 30663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } }
I20260928 05:28:17.371724 30663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:17.371809 30663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483ae408a4b74a44976ae2018f331ba0, State: Initialized, Role: FOLLOWER
I20260928 05:28:17.372404 30663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } }
I20260928 05:28:17.372615 30663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:17.372735 30663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:17.372885 30663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:17.374971 30663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } }
I20260928 05:28:17.375440 30663 leader_election.cc:302] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 483ae408a4b74a44976ae2018f331ba0; no voters:
I20260928 05:28:17.375955 30663 leader_election.cc:288] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:17.376065 30673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:17.377065 30673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [term 1 LEADER]: Becoming Leader. State: Replica: 483ae408a4b74a44976ae2018f331ba0, State: Running, Role: LEADER
I20260928 05:28:17.377543 30673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } }
I20260928 05:28:17.377923 30663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:17.380225 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "483ae408a4b74a44976ae2018f331ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } } }
I20260928 05:28:17.380208 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 483ae408a4b74a44976ae2018f331ba0. Latest consensus state: current_term: 1 leader_uuid: "483ae408a4b74a44976ae2018f331ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483ae408a4b74a44976ae2018f331ba0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 41379 } } }
I20260928 05:28:17.380648 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:17.380649 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:17.382541 30679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:17.386593 30679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:17.394780 30679 catalog_manager.cc:1347] Generated new cluster ID: 5b749366fd6d4e5cbb18fb16fc39bab3
I20260928 05:28:17.394883 30679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:17.411887 30679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:17.412322 30679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:17.418226 30679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 483ae408a4b74a44976ae2018f331ba0: Generated new TSK 0
I20260928 05:28:17.419023 30679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:17.424119 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30596
W20260928 05:28:17.425314 30658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33755
W20260928 05:28:17.426040 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30596: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:17.426158 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30596
2026-09-28T05:28:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1720 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:17 dist-test-slave-515x krb5kdc[30694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:17 dist-test-slave-515x krb5kdc[30694](info): set up 2 sockets
Sep 28 05:28:17 dist-test-slave-515x krb5kdc[30694](info): commencing operation
krb5kdc: starting...
W20260928 05:28:18.433813 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:18 dist-test-slave-515x krb5kdc[30694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:18.671118 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40087
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:40087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:18.984171 30707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:18.984443 30707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:18.984540 30707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:18.991575 30707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:18.991678 30707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:18.991752 30707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:18.991842 30707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:18.991899 30707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:19.001288 30707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:40087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:19.002534 30707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:19.003783 30707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:19.014832 30713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:19.014885 30712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:19.014885 30715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:19.015803 30707 server_base.cc:1034] running on GCE node
I20260928 05:28:19.016585 30707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:19.017997 30707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:19.019220 30707 hybrid_clock.cc:648] HybridClock initialized: now 1790573299019175 us; error 39 us; skew 500 ppm
I20260928 05:28:19.019553 30707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:19 dist-test-slave-515x krb5kdc[30694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573299, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:19.022564 30707 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:19.023984 30707 webserver.cc:460] Webserver started at http://127.8.124.254:43743/ using document root <none> and password file <none>
I20260928 05:28:19.024612 30707 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:19.024715 30707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:19.024956 30707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:19.027886 30707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "8e7ccf9196364b13aa0f42355f0736aa"
format_stamp: "Formatted at 2026-09-28 05:28:19 on dist-test-slave-515x"
server_key: "8ce054c28256d32a1e533040c68ac81c"
server_key_iv: "ce80514fe30f44fb297b99cfdd7345c4"
server_key_version: "encryptionkey@0"
I20260928 05:28:19.028592 30707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "8e7ccf9196364b13aa0f42355f0736aa"
format_stamp: "Formatted at 2026-09-28 05:28:19 on dist-test-slave-515x"
server_key: "8ce054c28256d32a1e533040c68ac81c"
server_key_iv: "ce80514fe30f44fb297b99cfdd7345c4"
server_key_version: "encryptionkey@0"
I20260928 05:28:19.033669 30707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 05:28:19.037009 30722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:19.038352 30707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:28:19.038493 30707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "8e7ccf9196364b13aa0f42355f0736aa"
format_stamp: "Formatted at 2026-09-28 05:28:19 on dist-test-slave-515x"
server_key: "8ce054c28256d32a1e533040c68ac81c"
server_key_iv: "ce80514fe30f44fb297b99cfdd7345c4"
server_key_version: "encryptionkey@0"
I20260928 05:28:19.038658 30707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:19.074338 30707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:19.078727 30707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:19.078984 30707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:19.096088 30707 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:40087
I20260928 05:28:19.096084 30773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:40087 every 8 connection(s)
I20260928 05:28:19.097457 30707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:19.101132 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:19.103829 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30707
I20260928 05:28:19.103998 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:19.104545 8691 external_mini_cluster.cc:1428] Setting key a6ca7ee8a87cf90034791a6aeca0e236
I20260928 05:28:19.112533 30774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa: Bootstrap starting.
I20260928 05:28:19.115657 30774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:19.116607 30774 log.cc:824] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:28:19 dist-test-slave-515x krb5kdc[30694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:19.119724 30774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa: No bootstrap required, opened a new log
I20260928 05:28:19.126112 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:19.106433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51897 (local address 127.8.124.254:40087)
0928 05:28:19.107226 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:19.107238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:19.107279 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:28:19.108882 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:19.108905 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:19.109285 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:19.109683 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:19.111069 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:19.112328 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:19.113637 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:19.114318 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:19.114477 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:19.118670 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:19.118678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:19.118692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:19.118741 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:19.121925 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:19.122766 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:19.122772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:19.122781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:19.122954 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:19.123353 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:19.123356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:19.123359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:19.123909 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:19.124148 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:19.124641 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 05:28:19.125012 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":344,"threads_started":1}
I20260928 05:28:19.128156 30774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } }
I20260928 05:28:19.128525 30774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:19.128624 30774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7ccf9196364b13aa0f42355f0736aa, State: Initialized, Role: FOLLOWER
I20260928 05:28:19.129243 30774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } }
I20260928 05:28:19.129468 30774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:19.129657 30774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:19.129757 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:19.129838 30774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:19.132894 30774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } }
I20260928 05:28:19.133507 30774 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7ccf9196364b13aa0f42355f0736aa; no voters:
I20260928 05:28:19.134123 30774 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:19.134194 30783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:19.135468 30783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [term 1 LEADER]: Becoming Leader. State: Replica: 8e7ccf9196364b13aa0f42355f0736aa, State: Running, Role: LEADER
I20260928 05:28:19.136096 30783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } }
I20260928 05:28:19.136333 30774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:19.138907 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7ccf9196364b13aa0f42355f0736aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } } }
I20260928 05:28:19.139411 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:19.139027 30785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7ccf9196364b13aa0f42355f0736aa. Latest consensus state: current_term: 1 leader_uuid: "8e7ccf9196364b13aa0f42355f0736aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7ccf9196364b13aa0f42355f0736aa" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 40087 } } }
I20260928 05:28:19.139989 30785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:19.150364 30791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:19.153752 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:19.134925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43847 (local address 127.8.124.254:40087)
0928 05:28:19.135093 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:19.135099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:19.136164 (+ 1065us) server_negotiation.cc:407] Connection header received
0928 05:28:19.136432 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:19.136437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:19.136519 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:19.136618 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:19.139180 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:19.140300 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:19.142298 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:19.143010 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:19.143131 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:19.146503 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:19.146511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:19.146517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:19.146557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:19.150325 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:19.151157 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:19.151164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:19.151168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:19.151342 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:19.151771 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:19.151778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:19.151782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:19.152192 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:19.152456 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:19.153054 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 05:28:19.153306 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:28:19.155521 30791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:19.165825 30791 catalog_manager.cc:1347] Generated new cluster ID: 788dd6b4af024e0b8d9af39091edf513
I20260928 05:28:19.166193 30791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:19.189883 30791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:19.191164 30791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:19.197956 30791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7ccf9196364b13aa0f42355f0736aa: Generated new TSK 0
I20260928 05:28:19.198998 30791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:19.206873 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30707
I20260928 05:28:19.208166 30769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:19.298180 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30707
2026-09-28T05:28:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:19 dist-test-slave-515x krb5kdc[30806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:19 dist-test-slave-515x krb5kdc[30806](info): set up 2 sockets
Sep 28 05:28:19 dist-test-slave-515x krb5kdc[30806](info): commencing operation
krb5kdc: starting...
W20260928 05:28:20.301712 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:20 dist-test-slave-515x krb5kdc[30806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:20.534649 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45379
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:20.848526 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:20.848836 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:20.848956 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:20.856294 30819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:20.856413 30819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:20.856474 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:20.856563 30819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:20.856616 30819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:20.866648 30819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:20.867962 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:20.869257 30819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:20.880498 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:20.880596 30824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:20.880561 30825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:20.881192 30819 server_base.cc:1034] running on GCE node
I20260928 05:28:20.881976 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:20.883391 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:20.884590 30819 hybrid_clock.cc:648] HybridClock initialized: now 1790573300884538 us; error 47 us; skew 500 ppm
I20260928 05:28:20.884924 30819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:20 dist-test-slave-515x krb5kdc[30806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573300, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:20.887952 30819 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:20.889448 30819 webserver.cc:460] Webserver started at http://127.8.124.254:37189/ using document root <none> and password file <none>
I20260928 05:28:20.890107 30819 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:20.890220 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:20.890494 30819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:20.893392 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "a377cb40ae6d4bed8e7b8ec869d2d706"
format_stamp: "Formatted at 2026-09-28 05:28:20 on dist-test-slave-515x"
server_key: "4d756542479830131c83b21c07f4f1fc"
server_key_iv: "6a3fee49b93defa4239bd4778d0cbb5b"
server_key_version: "encryptionkey@0"
I20260928 05:28:20.894122 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "a377cb40ae6d4bed8e7b8ec869d2d706"
format_stamp: "Formatted at 2026-09-28 05:28:20 on dist-test-slave-515x"
server_key: "4d756542479830131c83b21c07f4f1fc"
server_key_iv: "6a3fee49b93defa4239bd4778d0cbb5b"
server_key_version: "encryptionkey@0"
I20260928 05:28:20.899261 30819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:28:20.902563 30834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:20.903815 30819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:20.903972 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "a377cb40ae6d4bed8e7b8ec869d2d706"
format_stamp: "Formatted at 2026-09-28 05:28:20 on dist-test-slave-515x"
server_key: "4d756542479830131c83b21c07f4f1fc"
server_key_iv: "6a3fee49b93defa4239bd4778d0cbb5b"
server_key_version: "encryptionkey@0"
I20260928 05:28:20.904146 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:20.920729 30819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:20.925530 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:20.925853 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:20.941910 30819 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:45379
I20260928 05:28:20.941921 30885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:45379 every 8 connection(s)
I20260928 05:28:20.943248 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:20.945703 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30819
I20260928 05:28:20.945848 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:20.946277 8691 external_mini_cluster.cc:1428] Setting key 675f4f686db21a3936a998362ddedbd6
I20260928 05:28:20.947191 30886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:20.961539 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:20.948098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57187 (local address 127.8.124.254:45379)
0928 05:28:20.948967 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:20.948983 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:20.949335 (+ 352us) server_negotiation.cc:407] Connection header received
0928 05:28:20.951591 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:20.951619 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:20.952163 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:20.952604 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:20.953122 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:20.955163 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:20.956476 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:20.957501 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:20.957703 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:20.958814 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:20.958835 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:20.958855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:20.959566 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:20.959665 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:20.959695 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:20.959797 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:20.960012 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 05:28:20.960273 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":364,"threads_started":1}
I20260928 05:28:20.961591 30886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706: Bootstrap starting.
I20260928 05:28:20.965162 30886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:20.965762 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:20.966454 30886 log.cc:824] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:20.974742 30886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706: No bootstrap required, opened a new log
I20260928 05:28:20.977701 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:20.971187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46903 (local address 127.8.124.254:45379)
0928 05:28:20.971379 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:20.971385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:20.971801 (+ 416us) server_negotiation.cc:407] Connection header received
0928 05:28:20.971986 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:20.971992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:20.972071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:20.972154 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:20.972644 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:20.973719 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:20.975238 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:20.975679 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:20.975802 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:20.976833 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:20.976841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:20.976845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:20.976864 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:20.976905 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:20.976920 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:20.977004 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:20.977153 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 05:28:20.977252 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:28:20.982379 30886 raft_consensus.cc:348] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } }
I20260928 05:28:20.982777 30886 raft_consensus.cc:374] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:20.982869 30886 raft_consensus.cc:729] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a377cb40ae6d4bed8e7b8ec869d2d706, State: Initialized, Role: FOLLOWER
I20260928 05:28:20.983428 30886 consensus_queue.cc:260] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } }
I20260928 05:28:20.983645 30886 raft_consensus.cc:388] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:20.983775 30886 raft_consensus.cc:482] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:20.983927 30886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:20.986279 30886 raft_consensus.cc:504] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } }
I20260928 05:28:20.986754 30886 leader_election.cc:302] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a377cb40ae6d4bed8e7b8ec869d2d706; no voters:
I20260928 05:28:20.987332 30886 leader_election.cc:288] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:20.987468 30896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:20.988548 30896 raft_consensus.cc:686] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [term 1 LEADER]: Becoming Leader. State: Replica: a377cb40ae6d4bed8e7b8ec869d2d706, State: Running, Role: LEADER
I20260928 05:28:20.989090 30896 consensus_queue.cc:237] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } }
I20260928 05:28:20.989475 30886 sys_catalog.cc:564] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:20.991890 30898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a377cb40ae6d4bed8e7b8ec869d2d706. Latest consensus state: current_term: 1 leader_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } } }
I20260928 05:28:20.991936 30897 sys_catalog.cc:455] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a377cb40ae6d4bed8e7b8ec869d2d706" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 45379 } } }
I20260928 05:28:20.992319 30898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:20.992323 30897 sys_catalog.cc:458] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:20.993155 30906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:20.998322 30906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:21.008769 30906 catalog_manager.cc:1347] Generated new cluster ID: d1cbf5a111e74fd4ac18c1b0229b887e
I20260928 05:28:21.008898 30906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:21.036994 30906 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:21.038487 30906 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:21.048610 30906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a377cb40ae6d4bed8e7b8ec869d2d706: Generated new TSK 0
I20260928 05:28:21.049388 30906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:21.069872 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30819
W20260928 05:28:21.070914 30881 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57187
W20260928 05:28:21.071535 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30819: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:21.071656 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30819
2026-09-28T05:28:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:21 dist-test-slave-515x krb5kdc[30917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:21 dist-test-slave-515x krb5kdc[30917](info): set up 2 sockets
Sep 28 05:28:21 dist-test-slave-515x krb5kdc[30917](info): commencing operation
krb5kdc: starting...
W20260928 05:28:22.077808 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:22 dist-test-slave-515x krb5kdc[30917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:22.338944 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36537
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:36537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:22.652920 30930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:22.653185 30930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:22.653286 30930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:22.660547 30930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:22.660657 30930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:22.660705 30930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:22.660765 30930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:22.660808 30930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:22.670732 30930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:36537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:36537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:22.671947 30930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:22.673158 30930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:22.684873 30938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:22.684844 30936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:22.684918 30935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:22.685915 30930 server_base.cc:1034] running on GCE node
I20260928 05:28:22.686771 30930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:22.688169 30930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:22.689388 30930 hybrid_clock.cc:648] HybridClock initialized: now 1790573302689326 us; error 56 us; skew 500 ppm
I20260928 05:28:22.689771 30930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:22 dist-test-slave-515x krb5kdc[30917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:22.697201 30930 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:22.698726 30930 webserver.cc:460] Webserver started at http://127.8.124.254:32809/ using document root <none> and password file <none>
I20260928 05:28:22.699394 30930 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:22.699494 30930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:22.699748 30930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:22.702797 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "b82ed692e61f4b00a5def4e8f81808d0"
format_stamp: "Formatted at 2026-09-28 05:28:22 on dist-test-slave-515x"
server_key: "e8a2c4309c32038e50622691b8d35525"
server_key_iv: "898a19814ca1d3bed4f388cba4421cb9"
server_key_version: "encryptionkey@0"
I20260928 05:28:22.703526 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "b82ed692e61f4b00a5def4e8f81808d0"
format_stamp: "Formatted at 2026-09-28 05:28:22 on dist-test-slave-515x"
server_key: "e8a2c4309c32038e50622691b8d35525"
server_key_iv: "898a19814ca1d3bed4f388cba4421cb9"
server_key_version: "encryptionkey@0"
I20260928 05:28:22.708534 30930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:28:22.711987 30945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:22.713266 30930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:28:22.713426 30930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "b82ed692e61f4b00a5def4e8f81808d0"
format_stamp: "Formatted at 2026-09-28 05:28:22 on dist-test-slave-515x"
server_key: "e8a2c4309c32038e50622691b8d35525"
server_key_iv: "898a19814ca1d3bed4f388cba4421cb9"
server_key_version: "encryptionkey@0"
I20260928 05:28:22.713594 30930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:22.719040 30930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:22.722733 30930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:22.722976 30930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:22.739476 30930 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:36537
I20260928 05:28:22.739495 30996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:36537 every 8 connection(s)
I20260928 05:28:22.740831 30930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:22.744342 30997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:22.750964 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 30930
I20260928 05:28:22.751128 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:22.751561 8691 external_mini_cluster.cc:1428] Setting key c288ee1ab61829a47a480cbb92f97f0f
I20260928 05:28:22.754361 30997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0: Bootstrap starting.
I20260928 05:28:22.757561 30997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:22.758755 30997 log.cc:824] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:22.759094 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:22.753298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59667 (local address 127.8.124.254:36537)
0928 05:28:22.754087 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:22.754098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:22.754136 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:28:22.755808 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:22.755831 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:22.756165 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:22.756461 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:22.757513 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:22.757520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:22.757533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:22.757551 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:22.757642 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:22.757679 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:22.757829 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:22.757973 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 05:28:22.758159 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":361,"threads_started":1}
I20260928 05:28:22.761466 30997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0: No bootstrap required, opened a new log
I20260928 05:28:22.762633 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:22.769872 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:22.767429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46705 (local address 127.8.124.254:36537)
0928 05:28:22.767615 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:22.767622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:22.768034 (+ 412us) server_negotiation.cc:407] Connection header received
0928 05:28:22.768180 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:22.768185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:22.768235 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:22.768313 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:22.769216 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:22.769220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:22.769223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:28:22.769237 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:22.769278 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:28:22.769293 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:22.769403 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:22.769511 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 05:28:22.769561 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:28:22.771183 30997 raft_consensus.cc:348] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } }
I20260928 05:28:22.771508 30997 raft_consensus.cc:374] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:28:22.771595 30997 raft_consensus.cc:729] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82ed692e61f4b00a5def4e8f81808d0, State: Initialized, Role: FOLLOWER
I20260928 05:28:22.772130 30997 consensus_queue.cc:260] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } }
I20260928 05:28:22.772347 30997 raft_consensus.cc:388] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:28:22.772468 30997 raft_consensus.cc:482] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:28:22.772615 30997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:22.774773 30997 raft_consensus.cc:504] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } }
I20260928 05:28:22.775233 30997 leader_election.cc:302] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b82ed692e61f4b00a5def4e8f81808d0; no voters:
I20260928 05:28:22.775733 30997 leader_election.cc:288] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:28:22.775828 31007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:22.776449 31007 raft_consensus.cc:686] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [term 1 LEADER]: Becoming Leader. State: Replica: b82ed692e61f4b00a5def4e8f81808d0, State: Running, Role: LEADER
I20260928 05:28:22.776919 31007 consensus_queue.cc:237] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } }
I20260928 05:28:22.777590 30997 sys_catalog.cc:564] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:22.779475 31009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b82ed692e61f4b00a5def4e8f81808d0. Latest consensus state: current_term: 1 leader_uuid: "b82ed692e61f4b00a5def4e8f81808d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } } }
I20260928 05:28:22.780014 31009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:22.780748 31008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b82ed692e61f4b00a5def4e8f81808d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82ed692e61f4b00a5def4e8f81808d0" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 36537 } } }
I20260928 05:28:22.781064 31008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:22.781637 31012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:22.786995 31012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:22.795686 31012 catalog_manager.cc:1347] Generated new cluster ID: 3e3996a4d6d74326a02c143dd7a31803
I20260928 05:28:22.795797 31012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:22.833878 31012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:22.834306 31012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:22.839416 31012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b82ed692e61f4b00a5def4e8f81808d0: Generated new TSK 0
I20260928 05:28:22.840386 31012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:22.847124 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30930
W20260928 05:28:22.848192 30992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59667
W20260928 05:28:22.848883 8691 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 30930: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:28:22.849027 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 30930
2026-09-28T05:28:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1777 ms)
[----------] 18 tests from AuthTokenIssuingTest (32827 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:22 dist-test-slave-515x krb5kdc[31028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:22 dist-test-slave-515x krb5kdc[31028](info): set up 2 sockets
Sep 28 05:28:22 dist-test-slave-515x krb5kdc[31028](info): commencing operation
krb5kdc: starting...
W20260928 05:28:23.873979 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:24 dist-test-slave-515x krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:24.112848 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33021
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:24.432785 31041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:24.433101 31041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:24.433205 31041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:24.433300 31041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:24.440582 31041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:24.440671 31041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:24.440743 31041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:24.440811 31041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:24.440879 31041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:24.450811 31041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:24.452083 31041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:24.453320 31041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:24.464399 31046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:24.464401 31049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:24.464345 31047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:24.465210 31041 server_base.cc:1034] running on GCE node
I20260928 05:28:24.465943 31041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:24.467312 31041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:24.468535 31041 hybrid_clock.cc:648] HybridClock initialized: now 1790573304468475 us; error 35 us; skew 500 ppm
I20260928 05:28:24.468863 31041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:24 dist-test-slave-515x krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:24.471976 31041 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:24.473417 31041 webserver.cc:460] Webserver started at http://127.8.124.254:41721/ using document root <none> and password file <none>
I20260928 05:28:24.474102 31041 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:24.474191 31041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:24.474478 31041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:24.477347 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "f045c330d8364720b1b3ab50209391e5"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "2c29d2ae412ac9c4863805bee0762eec"
server_key_iv: "1f4e6639f3b7d41980c481251bf1a0f1"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.478155 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "f045c330d8364720b1b3ab50209391e5"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "2c29d2ae412ac9c4863805bee0762eec"
server_key_iv: "1f4e6639f3b7d41980c481251bf1a0f1"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.483289 31041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:28:24.486799 31056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:24.488153 31041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:28:24.488327 31041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "f045c330d8364720b1b3ab50209391e5"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "2c29d2ae412ac9c4863805bee0762eec"
server_key_iv: "1f4e6639f3b7d41980c481251bf1a0f1"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.488536 31041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:24.516724 31041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:24.521751 31041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:24.522037 31041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:24.539707 31041 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:33021
I20260928 05:28:24.539716 31107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:33021 every 8 connection(s)
I20260928 05:28:24.541038 31041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:24.544772 31108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:24.547016 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31041
I20260928 05:28:24.547263 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:24.548032 8691 external_mini_cluster.cc:1428] Setting key 0603f8846b00e3eeac122f94ca5c04c6
I20260928 05:28:24.553220 31108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:24 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.253@KRBTEST.COM" created.
I20260928 05:28:24.577658 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.556910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.8.124.254:33021)
0928 05:28:24.558689 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:24.558695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:24.558744 (+ 49us) server_negotiation.cc:407] Connection header received
0928 05:28:24.560442 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:24.560450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:24.560749 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:24.560940 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:24.562104 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:24.563808 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:24.565249 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:24.566062 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:24.566219 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:24.569672 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:24.569682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:24.569698 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:24.569743 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:24.573483 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:24.574227 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:24.574236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:24.574239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:24.574425 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:24.574876 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:24.574879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:24.574881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:24.575455 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:24.575609 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:24.576192 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 05:28:24.576470 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"spinlock_wait_cycles":73088,"thread_start_us":1373,"threads_started":1}
I20260928 05:28:24.577678 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.556910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33021 (local address 127.0.0.1:36222)
0928 05:28:24.557815 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.557861 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:24.560071 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:24.560973 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:24.560978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:24.561527 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:24.561901 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:24.561919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:24.563984 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:24.563989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:24.565110 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:24.565123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:24.566328 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:24.566334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:24.566462 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:24.567100 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:24.567133 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:24.569412 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:24.573670 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:24.573680 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:24.573689 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:24.574093 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:24.574574 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:24.574578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:24.574585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:24.574759 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:24.575600 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:24.575606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:24.575812 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 05:28:24.576028 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 05:28:24.576608 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":123,"thread_start_us":358,"threads_started":1}
I20260928 05:28:24.580687 31108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:24.581825 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.581407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53472)
0928 05:28:24.581569 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.581704 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 05:28:24.582166 31059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
I20260928 05:28:24.583616 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.583293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53484)
0928 05:28:24.583414 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.583482 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 05:28:24.583861 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:33903: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
I20260928 05:28:24.606500 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:33903
--webserver_interface=127.8.124.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:28:24.633165 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } attempt: 1
I20260928 05:28:24.634331 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.633882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53486)
0928 05:28:24.634068 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.634193 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 05:28:24.635269 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53488)
0928 05:28:24.635066 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.635137 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:28:24.635438 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:33903: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
I20260928 05:28:24.703696 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } attempt: 2
I20260928 05:28:24.704857 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.704382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53496)
0928 05:28:24.704619 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.704721 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:28:24.705842 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.705459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53512)
0928 05:28:24.705639 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.705713 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:28:24.706008 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:33903: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
I20260928 05:28:24.797256 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } attempt: 3
I20260928 05:28:24.798422 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.797969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53514)
0928 05:28:24.798155 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.798283 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:28:24.799407 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.799045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53516)
0928 05:28:24.799192 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.799270 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:28:24.799595 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:33903: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
W20260928 05:28:24.924845 31115 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:24.925179 31115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:24.925271 31115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:24.925395 31115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:24.932686 31115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:24.932781 31115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:24.932883 31115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:24.932943 31115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:24.933027 31115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:24.942746 31115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.124.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:24.942824 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } attempt: 4
I20260928 05:28:24.943910 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.943391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53528)
0928 05:28:24.943683 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.943776 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 05:28:24.943998 31115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:24.944839 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:24.944503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53532)
0928 05:28:24.944634 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:24.944708 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:28:24.944998 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:33903: Network error: Client connection negotiation failed: client connection to 127.8.124.253:33903: connect: Connection refused (error 111)
I20260928 05:28:24.945365 31115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:24.956372 31120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:24.956408 31123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:24.956431 31121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:24.957322 31115 server_base.cc:1034] running on GCE node
I20260928 05:28:24.958065 31115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:24.959370 31115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:24.960599 31115 hybrid_clock.cc:648] HybridClock initialized: now 1790573304960537 us; error 36 us; skew 500 ppm
I20260928 05:28:24.960888 31115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:24 dist-test-slave-515x krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:24.963909 31115 init.cc:377] Logged in from keytab as kudu/127.8.124.253@KRBTEST.COM (short username kudu)
I20260928 05:28:24.965278 31115 webserver.cc:460] Webserver started at http://127.8.124.253:38953/ using document root <none> and password file <none>
I20260928 05:28:24.965972 31115 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:24.966060 31115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:24.966349 31115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:24.969216 31115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data/instance:
uuid: "61923d035e804396be6e11f663f9cb74"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "190abb24970bb305b26b764112409fad"
server_key_iv: "34086c500eeb8485d69449c9946a0ec8"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.969986 31115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal/instance:
uuid: "61923d035e804396be6e11f663f9cb74"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "190abb24970bb305b26b764112409fad"
server_key_iv: "34086c500eeb8485d69449c9946a0ec8"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.974959 31115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:28:24.978242 31130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:24.979584 31115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:24.979732 31115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "61923d035e804396be6e11f663f9cb74"
format_stamp: "Formatted at 2026-09-28 05:28:24 on dist-test-slave-515x"
server_key: "190abb24970bb305b26b764112409fad"
server_key_iv: "34086c500eeb8485d69449c9946a0ec8"
server_key_version: "encryptionkey@0"
I20260928 05:28:24.979923 31115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:25.008916 31115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:25.013234 31115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:25.013505 31115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:25.030303 31115 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.253:33903
I20260928 05:28:25.030328 31181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.253:33903 every 8 connection(s)
I20260928 05:28:25.031629 31115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
I20260928 05:28:25.035284 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:25.039932 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31115
I20260928 05:28:25.040105 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-1/wal/instance
I20260928 05:28:25.040625 8691 external_mini_cluster.cc:1428] Setting key 3320910ebd21992f98415c6b386ab587
I20260928 05:28:25.042853 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:25.063745 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.045805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.8.124.254:33021)
0928 05:28:25.046008 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.046015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.047230 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 05:28:25.049486 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.049490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.049556 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.049676 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.051316 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.052293 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.053778 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.054339 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.054453 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.057927 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.057932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.057935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.057969 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.060516 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.061371 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.061376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.061380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.061497 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.061964 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.061967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.061970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.062290 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.062411 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.063328 (+ 917us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.063519 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:28:25.065070 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.046303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33021 (local address 127.0.0.1:36230)
0928 05:28:25.047042 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.047095 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.049224 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.049883 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.049893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.050565 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.051136 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.051153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.052447 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.052451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.053645 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.053659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.054525 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.054530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.054734 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.055423 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.055444 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.057713 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.060649 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.060656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.060671 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.061241 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.061595 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.061621 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.061625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.061851 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.062418 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.062426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.062875 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.063172 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.063710 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
WARNING: no policy specified for kudu/127.8.124.252@KRBTEST.COM; defaulting to no policy
I20260928 05:28:25.066780 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.252@KRBTEST.COM" created.
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:25.080669 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53546)
0928 05:28:25.067637 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.067652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.067769 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.068748 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.068753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.069149 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.069285 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.069293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.070640 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.070645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.071464 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.071475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.072237 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.072243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.072404 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.072923 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.072939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.074963 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.078518 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.078522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.078525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.078860 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.079206 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.079209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.079211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.079340 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.079953 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.079957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.080066 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.080173 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.080403 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:28:25.080909 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.067708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.8.124.253:33903)
0928 05:28:25.068363 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.068368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.068392 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:28:25.068432 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.068438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.068605 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.068740 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.069437 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.070518 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.071586 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.072089 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.072206 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.075178 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.075191 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.075215 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.075249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.078402 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.078979 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.078987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.078990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.079116 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.079426 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.079430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.079433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.079832 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.080017 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.080299 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.080521 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":348,"threads_started":1}
I20260928 05:28:25.083400 31182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:25.084584 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.084096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53302)
0928 05:28:25.084307 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.084445 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 05:28:25.084853 31134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.086346 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.086035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53304)
0928 05:28:25.086166 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.086245 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 05:28:25.086555 31182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
I20260928 05:28:25.100795 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:41929
--webserver_interface=127.8.124.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:28:25.135861 31182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 1
I20260928 05:28:25.137077 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.136551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53312)
0928 05:28:25.136760 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.136889 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 05:28:25.138121 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.137747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53318)
0928 05:28:25.137896 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.137988 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 05:28:25.138363 31182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.206594 31182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 2
I20260928 05:28:25.207828 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.207347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53330)
0928 05:28:25.207559 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.207653 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 05:28:25.208894 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.208524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53342)
0928 05:28:25.208687 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.208767 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 05:28:25.209103 31182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.244246 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } attempt: 5
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:25.256976 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.244914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53552)
0928 05:28:25.245101 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.245123 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.245241 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.245649 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.245655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.246028 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.246206 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.246216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.247197 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.247203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.248045 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.248065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.248768 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.248775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.248982 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.249659 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.249681 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.251631 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.254641 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.254647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.254651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.255036 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.255385 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.255390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.255394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.255539 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.256044 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.256048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.256210 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.256338 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.256616 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 05:28:25.256980 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.245123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.8.124.253:33903)
0928 05:28:25.245312 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.245317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.245335 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:28:25.245395 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.245398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.245470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.245553 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.246357 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.247064 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.248199 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.248628 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.248738 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.251808 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.251813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.251816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.251848 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.254508 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.255148 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.255153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.255157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.255278 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.255645 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.255649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.255652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.255927 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.256057 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.256453 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.256681 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:28:25.258134 31108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:25.259336 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.258963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53348)
0928 05:28:25.259123 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.259197 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:28:25.260381 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.260007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53364)
0928 05:28:25.260187 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.260269 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:28:25.260576 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.300311 31182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 3
I20260928 05:28:25.301491 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.301008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53380)
0928 05:28:25.301234 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.301328 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 05:28:25.302526 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.302103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53396)
0928 05:28:25.302327 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.302406 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260928 05:28:25.302742 31182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.311795 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 1
I20260928 05:28:25.312808 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.312368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53406)
0928 05:28:25.312553 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.312643 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:28:25.313725 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.313380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53410)
0928 05:28:25.313517 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.313580 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:28:25.313913 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.382104 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 2
I20260928 05:28:25.383152 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.382693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53414)
0928 05:28:25.382904 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.382988 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:28:25.384054 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.383758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53424)
0928 05:28:25.383890 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.383955 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:28:25.384261 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
W20260928 05:28:25.420423 31189 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:25.420760 31189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:25.420855 31189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:25.420970 31189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:25.428031 31189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:25.428123 31189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:25.428198 31189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:25.428261 31189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:25.428344 31189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:25.438210 31189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:33021,127.8.124.253:33903,127.8.124.252:41929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:41929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.124.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:25.439473 31189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:25.440749 31189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 05:28:25.445940 31182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 4
I20260928 05:28:25.447109 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.446604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53436)
0928 05:28:25.446833 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.446935 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:28:25.447988 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.447680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53448)
0928 05:28:25.447816 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.447887 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:28:25.448185 31182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
W20260928 05:28:25.452207 31194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:25.452138 31195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:25.452172 31197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:25.453013 31189 server_base.cc:1034] running on GCE node
I20260928 05:28:25.453802 31189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:25.455120 31189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:25.456353 31189 hybrid_clock.cc:648] HybridClock initialized: now 1790573305456289 us; error 36 us; skew 500 ppm
I20260928 05:28:25.456671 31189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573305, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:25.459602 31189 init.cc:377] Logged in from keytab as kudu/127.8.124.252@KRBTEST.COM (short username kudu)
I20260928 05:28:25.461109 31189 webserver.cc:460] Webserver started at http://127.8.124.252:37529/ using document root <none> and password file <none>
I20260928 05:28:25.461827 31189 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:25.461931 31189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:25.462242 31189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:25.465384 31189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data/instance:
uuid: "a9c142aaeaa14933969da6f56164dc42"
format_stamp: "Formatted at 2026-09-28 05:28:25 on dist-test-slave-515x"
server_key: "565daed34f72f5a8f7f70896af3fb88c"
server_key_iv: "103a06ba2e99de2e003277376d06d878"
server_key_version: "encryptionkey@0"
I20260928 05:28:25.466195 31189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal/instance:
uuid: "a9c142aaeaa14933969da6f56164dc42"
format_stamp: "Formatted at 2026-09-28 05:28:25 on dist-test-slave-515x"
server_key: "565daed34f72f5a8f7f70896af3fb88c"
server_key_iv: "103a06ba2e99de2e003277376d06d878"
server_key_version: "encryptionkey@0"
I20260928 05:28:25.471182 31189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:28:25.474684 31204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:25.476029 31189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:28:25.476191 31189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "a9c142aaeaa14933969da6f56164dc42"
format_stamp: "Formatted at 2026-09-28 05:28:25 on dist-test-slave-515x"
server_key: "565daed34f72f5a8f7f70896af3fb88c"
server_key_iv: "103a06ba2e99de2e003277376d06d878"
server_key_version: "encryptionkey@0"
I20260928 05:28:25.476406 31189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:25.490140 31189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:25.490456 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 3
I20260928 05:28:25.491508 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.491038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53460)
0928 05:28:25.491273 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.491360 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:28:25.492583 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.492216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53466)
0928 05:28:25.492399 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.492475 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 05:28:25.492812 31108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:41929: Network error: Client connection negotiation failed: client connection to 127.8.124.252:41929: connect: Connection refused (error 111)
I20260928 05:28:25.494580 31189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:25.494817 31189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:25.511404 31189 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.252:41929
I20260928 05:28:25.511422 31255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.252:41929 every 8 connection(s)
I20260928 05:28:25.512697 31189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
I20260928 05:28:25.513895 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31189
I20260928 05:28:25.514070 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572965871275-8691-0/minicluster-data/master-2/wal/instance
I20260928 05:28:25.514595 8691 external_mini_cluster.cc:1428] Setting key 7c7784f96558df82dddd22bc851592a6
I20260928 05:28:25.517165 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:25.525103 31256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573305, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:25.537214 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.516467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.8.124.252:41929)
0928 05:28:25.517230 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.517245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.517294 (+ 49us) server_negotiation.cc:407] Connection header received
0928 05:28:25.518898 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.518920 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.519304 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.519635 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.521398 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.522777 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.524520 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.525212 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.525357 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.530018 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.530026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.530035 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.530069 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.532870 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.533544 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.533550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.533556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.533750 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.534148 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.534152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.534154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.534697 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.534926 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.535490 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.535927 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":356,"threads_started":1}
I20260928 05:28:25.543447 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.528013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:33021 (local address 127.0.0.1:36236)
0928 05:28:25.528766 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.528809 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.529028 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.529393 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.529400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.530173 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.530461 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.530471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.531426 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.531430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.532567 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.532579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.533377 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.533383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.533545 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.534098 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.534122 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.536756 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.540241 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.540252 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.540259 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.540875 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.541355 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.541358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.541360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.541482 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.542283 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.542292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.542502 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.542712 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.543033 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":350,"threads_started":1}
I20260928 05:28:25.543483 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.528077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.8.124.254:33021)
0928 05:28:25.528300 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.528306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.528899 (+ 593us) server_negotiation.cc:407] Connection header received
0928 05:28:25.529157 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.529161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.529223 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.529302 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.530607 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.531285 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.532713 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.533131 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.533328 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.536973 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.536978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.536981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.537017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.540108 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.541035 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.541044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.541047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.541218 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.541628 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.541633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.541636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.542011 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.542172 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.542815 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.543051 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:28:25.545271 31256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573305, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:25.556666 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.540477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.8.124.254:33021)
0928 05:28:25.541289 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.541295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.541317 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:28:25.541364 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.541369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.541433 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.541526 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.543160 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.544410 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.545960 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.546576 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.546686 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.551333 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.551338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.551345 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.551379 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.554161 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.554810 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.554815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.554818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.554939 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.555295 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.555298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.555300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.555561 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.555810 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.556116 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.556345 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":476,"threads_started":1}
I20260928 05:28:25.559624 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.546102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:33903 (local address 127.0.0.1:53566)
0928 05:28:25.546373 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.546391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.546496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.546940 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.546944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.547497 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.547667 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.547676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.548618 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.548624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.549508 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.549519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.551302 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.551306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.551456 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.552094 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.552110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.554124 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.556947 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.556951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.556955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.557488 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.557933 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.557938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.557942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.558089 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.558797 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.558802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.558946 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.559055 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.559293 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 05:28:25.559700 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.546096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.8.124.253:33903)
0928 05:28:25.546264 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.546271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.546460 (+ 189us) server_negotiation.cc:407] Connection header received
0928 05:28:25.546605 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.546611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.546701 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.546789 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.547821 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.548483 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.549667 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.550326 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.550451 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.554311 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.554317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.554320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.554351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.556813 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.557645 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.557650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.557654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.557825 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.558236 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.558239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.558241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.558551 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.558720 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.559179 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.559423 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:28:25.560698 31256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573305, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:28:25.573155 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.558469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.8.124.253:33903)
0928 05:28:25.558995 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.558999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.559016 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:28:25.559055 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.559059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.559123 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.559224 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.560710 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.561865 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.563006 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.563429 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.563536 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.567750 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.567756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.567759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.567791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.570233 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.571027 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.571033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.571037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.571230 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.571718 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.571722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.571724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.572044 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.572323 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.572552 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.572760 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":265,"threads_started":1}
I20260928 05:28:25.573294 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.561513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53496)
0928 05:28:25.561740 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.561768 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.561888 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.562252 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.562255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.562581 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.562704 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.562711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.563605 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.563610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.564563 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.564575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.565236 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.565242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.565413 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.565926 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.565949 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.567960 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.570756 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.570761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.570765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.571223 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.571646 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.571651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.571655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.571796 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.572335 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.572340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.572491 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.572638 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.572913 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 05:28:25.573475 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.561491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.8.124.252:41929)
0928 05:28:25.561682 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.561687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.561829 (+ 142us) server_negotiation.cc:407] Connection header received
0928 05:28:25.562002 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.562007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.562060 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.562104 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.562830 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.563491 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.564689 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.565094 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.565197 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.568129 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.568135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.568138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.568165 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.570637 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.571351 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.571357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.571361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.571509 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.571897 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.571902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.571905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.572221 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.572346 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.572745 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.573097 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":56}
I20260928 05:28:25.578020 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:25.579093 31256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: Bootstrap starting.
I20260928 05:28:25.582335 31256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:25.580572 31083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f045c330d8364720b1b3ab50209391e5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36240
I20260928 05:28:25.583365 31256 log.cc:824] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:25.586427 31256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: No bootstrap required, opened a new log
I20260928 05:28:25.593483 31256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.593842 31256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c142aaeaa14933969da6f56164dc42, State: Initialized, Role: FOLLOWER
I20260928 05:28:25.594264 31256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.595153 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.595536 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.596040 31256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:28:25.604801 31275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:28:25.604974 31275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:28:25.602649 31189 master_runner.cc:186] Error getting master registration for 127.8.124.254:33021: OK, instance_id { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" instance_seqno: 1790573304534666 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:28:25.612646 31189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:28:25.670037 31108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 4
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:28:25.685914 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.670875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.8.124.252:41929)
0928 05:28:25.671114 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.671121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.671177 (+ 56us) server_negotiation.cc:407] Connection header received
0928 05:28:25.671386 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.671392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.671473 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.671593 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.672489 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.673660 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.675410 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.676087 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.676210 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.679972 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.679979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.679984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.680026 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.683240 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.683890 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.683896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.683900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.684046 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.684471 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.684476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.684480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.684809 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.684952 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.685322 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.685528 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:28:25.685910 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.670813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53506)
0928 05:28:25.671098 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.671117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.671263 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.671626 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.671631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.672120 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.672339 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.672351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.673779 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.673785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.675266 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.675283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.676229 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.676235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.676416 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.677092 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.677115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.679742 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.683368 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.683374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.683378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.683773 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.684145 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.684150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.684154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.684362 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.684920 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.684925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.685085 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.685202 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.685485 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 05:28:25.692011 31108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: Bootstrap starting.
I20260928 05:28:25.696091 31108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:25.697407 31108 log.cc:824] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:25.700901 31108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: No bootstrap required, opened a new log
I20260928 05:28:25.708680 31108 raft_consensus.cc:348] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.709151 31108 raft_consensus.cc:729] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f045c330d8364720b1b3ab50209391e5, State: Initialized, Role: FOLLOWER
I20260928 05:28:25.709734 31108 consensus_queue.cc:260] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.710788 31278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.711184 31278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.712127 31108 sys_catalog.cc:564] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:25.712275 31083 raft_consensus.cc:482] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:28:25.712471 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:25.714956 31083 raft_consensus.cc:504] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.717067 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f045c330d8364720b1b3ab50209391e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9c142aaeaa14933969da6f56164dc42"
I20260928 05:28:25.717593 31231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:25.720913 31231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f045c330d8364720b1b3ab50209391e5 in term 1.
I20260928 05:28:25.721577 31060 leader_election.cc:302] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9c142aaeaa14933969da6f56164dc42, f045c330d8364720b1b3ab50209391e5; no voters:
I20260928 05:28:25.722008 31083 leader_election.cc:288] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [CANDIDATE]: Term 1 election: Requested vote from peers 61923d035e804396be6e11f663f9cb74 (127.8.124.253:33903), a9c142aaeaa14933969da6f56164dc42 (127.8.124.252:41929)
I20260928 05:28:25.722038 31278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:25.735137 31278 raft_consensus.cc:686] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [term 1 LEADER]: Becoming Leader. State: Replica: f045c330d8364720b1b3ab50209391e5, State: Running, Role: LEADER
I20260928 05:28:25.736505 31278 consensus_queue.cc:237] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
W20260928 05:28:25.746799 31210 tablet.cc:2369] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:28:25.747368 31182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } attempt: 5
I20260928 05:28:25.751087 31283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f045c330d8364720b1b3ab50209391e5. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.751504 31283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: This master's current role is: LEADER
Sep 28 05:28:25 dist-test-slave-515x krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:28:25.768110 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.748811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.8.124.252:41929)
0928 05:28:25.748998 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.749004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.749025 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:28:25.749089 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.749094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.749173 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.749274 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.750486 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.751570 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.753443 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.754251 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.754372 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.760038 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.760044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.760049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.760095 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.764013 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.765338 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.765345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.765350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.765526 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.766282 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.766289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.766292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.766635 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.766761 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.767524 (+ 763us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.767761 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:28:25.770565 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.748033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:41929 (local address 127.0.0.1:53512)
0928 05:28:25.748433 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:25.748454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:25.748585 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:25.749497 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:25.749503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:25.750059 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:25.750287 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.750301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.751766 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.751772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.753255 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:25.753274 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.754589 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.754596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:25.754847 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.755652 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:25.755679 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:25.759791 (+ 4112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:25.764296 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.764303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.764308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.765131 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.765821 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:25.765826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:25.765830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:25.766088 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:25.766896 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:25.766901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:25.767143 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 05:28:25.768622 (+ 1479us) client_negotiation.cc:240] Negotiation successful
0928 05:28:25.769090 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
W20260928 05:28:25.774878 31062 tablet.cc:2369] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:28:25.776993 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: Bootstrap starting.
I20260928 05:28:25.778223 31290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:28:25.780896 31182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:25.781946 31182 log.cc:824] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:25.793790 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: No bootstrap required, opened a new log
I20260928 05:28:25.801350 31290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:25.813905 31182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.814235 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.768297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.8.124.253:33903)
0928 05:28:25.768924 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.768931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.789262 (+ 20331us) server_negotiation.cc:407] Connection header received
0928 05:28:25.792760 (+ 3498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.792767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.792853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.792923 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.795230 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.796381 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.801622 (+ 5241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.802258 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.802358 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.805664 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.805671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.805676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.805715 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.809332 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.810611 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.810616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.810620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.810764 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.811195 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.811199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.811203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.811891 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.812111 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.813619 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.813886 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469}
I20260928 05:28:25.814334 31182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61923d035e804396be6e11f663f9cb74, State: Initialized, Role: FOLLOWER
I20260928 05:28:25.815687 31182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } }
I20260928 05:28:25.823900 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.784213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.8.124.252:41929)
0928 05:28:25.784399 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.784405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.799042 (+ 14637us) server_negotiation.cc:407] Connection header received
0928 05:28:25.799538 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.799545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.799639 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.799707 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.801498 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.802532 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.811876 (+ 9344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.812665 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.812774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.816318 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.816325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.816331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.816378 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.820003 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.820886 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.820893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.820896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.821041 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.821688 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.821693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.821696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.822063 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.822294 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.823149 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.823389 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:28:25.824934 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:25.769075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.8.124.254:33021)
0928 05:28:25.769285 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:25.769292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:25.786531 (+ 17239us) server_negotiation.cc:407] Connection header received
0928 05:28:25.787051 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:25.787061 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:25.787173 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:25.787254 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:25.789564 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.790820 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.793424 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:25.794164 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:25.794263 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:25.810245 (+ 15982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:25.810267 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:25.810276 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:25.810322 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:25.821307 (+ 10985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.822141 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.822147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.822150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.822291 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:25.822746 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:25.822751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:25.822755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:25.823118 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:25.823336 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:25.824367 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0928 05:28:25.824586 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:28:25.827209 31302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.827891 31302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.828603 31182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:25.828640 31157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f045c330d8364720b1b3ab50209391e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61923d035e804396be6e11f663f9cb74"
I20260928 05:28:25.829093 31157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:25.832545 31157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f045c330d8364720b1b3ab50209391e5 in term 1.
I20260928 05:28:25.850044 31231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [term 1 FOLLOWER]: Refusing update from remote peer f045c330d8364720b1b3ab50209391e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:28:25.850258 31157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [term 1 FOLLOWER]: Refusing update from remote peer f045c330d8364720b1b3ab50209391e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:28:25.851239 31278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:28:25.852768 31283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:28:25.878991 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f045c330d8364720b1b3ab50209391e5. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.879343 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.882496 31278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.882807 31278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:25.883121 31303 mvcc.cc:205] Tried to move back new op lower bound from 7334188260745207808 to 7334188260309676032. Current Snapshot: MvccSnapshot[applied={T|T < 7334188260745207808}]
I20260928 05:28:25.827873 31041 master_runner.cc:186] Error getting master registration for 127.8.124.253:33903: OK, instance_id { permanent_uuid: "61923d035e804396be6e11f663f9cb74" instance_seqno: 1790573305025410 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:28:25.883652 31041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:28:25.885704 31305 mvcc.cc:205] Tried to move back new op lower bound from 7334188260745207808 to 7334188260309676032. Current Snapshot: MvccSnapshot[applied={T|T < 7334188260745207808}]
I20260928 05:28:25.887212 31290 catalog_manager.cc:1347] Generated new cluster ID: ea000a092c2a4b038d74c52405f8f8e0
I20260928 05:28:25.887301 31290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:25.887192 31302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f045c330d8364720b1b3ab50209391e5. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.887506 31302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.888803 31283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.889047 31283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:25.903203 31302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.903467 31302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.906117 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f045c330d8364720b1b3ab50209391e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f045c330d8364720b1b3ab50209391e5" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 33021 } } peers { permanent_uuid: "61923d035e804396be6e11f663f9cb74" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 33903 } } peers { permanent_uuid: "a9c142aaeaa14933969da6f56164dc42" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 41929 } } }
I20260928 05:28:25.906407 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:25.928081 31290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:25.930299 31290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:25.963405 31290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f045c330d8364720b1b3ab50209391e5: Generated new TSK 0
I20260928 05:28:25.964480 31290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:25.972637 31321 catalog_manager.cc:1259] Loaded cluster ID: ea000a092c2a4b038d74c52405f8f8e0
I20260928 05:28:25.972754 31321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: loading cluster ID for follower catalog manager: success
I20260928 05:28:25.998463 31321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: acquiring CA information for follower catalog manager: success
I20260928 05:28:26.006306 31321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61923d035e804396be6e11f663f9cb74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:28:26.100129 31262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.093120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.8.124.253:33903)
0928 05:28:26.093309 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.093315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.097637 (+ 4322us) server_negotiation.cc:407] Connection header received
0928 05:28:26.097968 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.097975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.098061 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.098132 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.099857 (+ 1725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53598: BlockingRecv error: recv got EOF from 127.0.0.1:53598 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:28:26.102591 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.094289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.8.124.252:41929)
0928 05:28:26.094445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.094450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.099395 (+ 4945us) server_negotiation.cc:407] Connection header received
0928 05:28:26.101140 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.101146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.101231 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.101299 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.102342 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53534: BlockingRecv error: recv got EOF from 127.0.0.1:53534 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 05:28:26.105113 31261 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.088659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.8.124.254:33021)
0928 05:28:26.089008 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.089015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.094160 (+ 5145us) server_negotiation.cc:407] Connection header received
0928 05:28:26.103594 (+ 9434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.103600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.103700 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.103769 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.104849 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36272: BlockingRecv error: recv got EOF from 127.0.0.1:36272 (error 108)
Metrics: {"server-negotiator.queue_time_us":217}
I20260928 05:28:26.150043 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.134613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.8.124.253:33903)
0928 05:28:26.134831 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.134838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.142436 (+ 7598us) server_negotiation.cc:407] Connection header received
0928 05:28:26.142941 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.142947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.143254 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.143396 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.144086 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.145186 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.147163 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.147819 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.147912 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.148618 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.149350 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.149475 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.149588 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.149706 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 05:28:26.160807 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.158805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.8.124.252:41929)
0928 05:28:26.158998 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.159004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.159465 (+ 461us) server_negotiation.cc:407] Connection header received
0928 05:28:26.159660 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.159664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.159761 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.159847 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.160567 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:28:26.169559 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.164479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.8.124.253:33903)
0928 05:28:26.164675 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.164681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.165130 (+ 449us) server_negotiation.cc:407] Connection header received
0928 05:28:26.165277 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.165282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.165348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.165420 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.165837 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.166555 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.167868 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.168324 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.168451 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.168836 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.168964 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.169036 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.169182 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.169262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 05:28:26.177404 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.175531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.8.124.252:41929)
0928 05:28:26.175726 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.175732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.176156 (+ 424us) server_negotiation.cc:407] Connection header received
0928 05:28:26.176296 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.176301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.176388 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.176470 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.177183 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53562: BlockingRecv error: recv got EOF from 127.0.0.1:53562 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:28:26.196755 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.191223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.8.124.253:33903)
0928 05:28:26.191443 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.191450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.192010 (+ 560us) server_negotiation.cc:407] Connection header received
0928 05:28:26.192222 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.192228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.192301 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.192384 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.192897 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.193748 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.195074 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.195508 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.195616 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.196029 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.196136 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.196218 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.196386 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.196470 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 05:28:26.204830 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.202731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.8.124.252:41929)
0928 05:28:26.202897 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.202901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.203602 (+ 701us) server_negotiation.cc:407] Connection header received
0928 05:28:26.203790 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.203794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.203866 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.203949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.204644 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:28:26.217686 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.212234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.8.124.253:33903)
0928 05:28:26.212472 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.212478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.213002 (+ 524us) server_negotiation.cc:407] Connection header received
0928 05:28:26.213201 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.213206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.213294 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.213376 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.213900 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.214688 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.215984 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.216437 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.216566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.216933 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.217072 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.217152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.217287 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.217367 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260928 05:28:26.225297 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.223274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.8.124.252:41929)
0928 05:28:26.223562 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.223568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.224047 (+ 479us) server_negotiation.cc:407] Connection header received
0928 05:28:26.224225 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.224230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.224306 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.224386 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.225081 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 05:28:26.241411 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.236065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.8.124.253:33903)
0928 05:28:26.236286 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.236293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.236762 (+ 469us) server_negotiation.cc:407] Connection header received
0928 05:28:26.236938 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.236944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.237034 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.237108 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.237522 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.238323 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.239660 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.240149 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.240254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.240637 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.240747 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.240819 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.240954 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.241038 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 05:28:26.248907 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.246804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.8.124.252:41929)
0928 05:28:26.246992 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.246999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.247642 (+ 643us) server_negotiation.cc:407] Connection header received
0928 05:28:26.247791 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.247798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.247874 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.247956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.248682 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53594: BlockingRecv error: recv got EOF from 127.0.0.1:53594 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:28:26.274154 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.268983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.8.124.253:33903)
0928 05:28:26.269169 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.269176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.269667 (+ 491us) server_negotiation.cc:407] Connection header received
0928 05:28:26.269867 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.269872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.269935 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.270008 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.270402 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.271159 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.272551 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.272963 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.273070 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.273442 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.273547 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.273659 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.273798 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.273875 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 05:28:26.283272 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.279724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.8.124.252:41929)
0928 05:28:26.279925 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.279929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.280575 (+ 646us) server_negotiation.cc:407] Connection header received
0928 05:28:26.280759 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.280764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.280842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.280947 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.282853 (+ 1906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53602: BlockingRecv error: recv got EOF from 127.0.0.1:53602 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 05:28:26.324452 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.319263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.8.124.253:33903)
0928 05:28:26.319468 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.319475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.319990 (+ 515us) server_negotiation.cc:407] Connection header received
0928 05:28:26.320182 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.320186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.320255 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.320331 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.320757 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.321506 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.322824 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.323226 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.323338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.323720 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.323845 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.323918 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.324061 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.324143 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 05:28:26.337003 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.334976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.8.124.252:41929)
0928 05:28:26.335203 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.335209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.335699 (+ 490us) server_negotiation.cc:407] Connection header received
0928 05:28:26.335870 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.335875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.335955 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.336042 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.336785 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53606: BlockingRecv error: recv got EOF from 127.0.0.1:53606 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:28:26.414036 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.408157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.8.124.253:33903)
0928 05:28:26.408387 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.408393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.408748 (+ 355us) server_negotiation.cc:407] Connection header received
0928 05:28:26.408926 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.408932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.409027 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.409144 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.409471 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.410660 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.412005 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.412697 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.412826 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.413241 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.413393 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.413495 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.413589 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.413691 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:28:26.429319 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.427346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.8.124.252:41929)
0928 05:28:26.427548 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.427555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.427876 (+ 321us) server_negotiation.cc:407] Connection header received
0928 05:28:26.428032 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.428037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.428139 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.428275 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.429060 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53614: BlockingRecv error: recv got EOF from 127.0.0.1:53614 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:28:26.565235 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.560271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.8.124.253:33903)
0928 05:28:26.560524 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.560538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.560870 (+ 332us) server_negotiation.cc:407] Connection header received
0928 05:28:26.561062 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.561066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.561140 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.561216 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.561673 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.562361 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.563682 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.564116 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.564224 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.564593 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.564700 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.564773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.564909 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.564985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 05:28:26.572299 31259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:26.570756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.8.124.252:41929)
0928 05:28:26.570900 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.570904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.571200 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:28:26.571312 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.571316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.571394 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.571478 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:26.572084 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53630: BlockingRecv error: recv got EOF from 127.0.0.1:53630 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:28:26.606964 31275 catalog_manager.cc:1259] Loaded cluster ID: ea000a092c2a4b038d74c52405f8f8e0
I20260928 05:28:26.607079 31275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: loading cluster ID for follower catalog manager: success
I20260928 05:28:26.609474 31275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: acquiring CA information for follower catalog manager: success
I20260928 05:28:26.611258 31275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9c142aaeaa14933969da6f56164dc42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:28:26.839983 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.835302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.8.124.253:33903)
0928 05:28:26.835476 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.835481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.835791 (+ 310us) server_negotiation.cc:407] Connection header received
0928 05:28:26.835957 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.835961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.836029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.836114 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.836460 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.837148 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.838514 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.838909 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.839009 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.839375 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.839474 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.839535 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.839664 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.839740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:28:26.850803 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.845282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.8.124.252:41929)
0928 05:28:26.845453 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.845459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.845841 (+ 382us) server_negotiation.cc:407] Connection header received
0928 05:28:26.846013 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.846017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.846174 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.846272 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.846662 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.847346 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.848632 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.849027 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.849133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.849638 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.850246 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.850378 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.850459 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.850540 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:28:26.864437 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.856875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.8.124.252:41929)
0928 05:28:26.857224 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.857230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.858400 (+ 1170us) server_negotiation.cc:407] Connection header received
0928 05:28:26.858605 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.858610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.858700 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.858765 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.859641 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.860416 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.862256 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.862819 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.862934 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.863375 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.863513 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.863572 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.864099 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.864184 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260928 05:28:26.864848 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.856693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.8.124.253:33903)
0928 05:28:26.857077 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.857083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.858919 (+ 1836us) server_negotiation.cc:407] Connection header received
0928 05:28:26.859054 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.859059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.859128 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.859208 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.859906 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.860951 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.862681 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.863324 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.863427 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.863982 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.864105 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.864185 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.864514 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.864592 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260928 05:28:26.865680 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:26.856742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.8.124.254:33021)
0928 05:28:26.857551 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:26.857557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:26.858228 (+ 671us) server_negotiation.cc:407] Connection header received
0928 05:28:26.858590 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:26.858595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:26.858806 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:26.858940 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:26.859793 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.860928 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.862868 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:26.863406 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:26.863511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:26.864071 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:26.864526 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:26.864612 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:26.865290 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 05:28:26.865370 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":612,"threads_started":1}
I20260928 05:28:26.870456 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31041
I20260928 05:28:26.871381 31103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:26.980356 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31041
W20260928 05:28:26.997411 31208 connection.cc:535] server connection from 127.0.0.1:53506 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 05:28:26.997843 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31115
I20260928 05:28:26.998919 31177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:27.105581 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31115
I20260928 05:28:27.121934 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31189
I20260928 05:28:27.123023 31251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:27.232000 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31189
2026-09-28T05:28:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4388 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:28:27 dist-test-slave-515x krb5kdc[31446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:28:27 dist-test-slave-515x krb5kdc[31446](info): set up 2 sockets
Sep 28 05:28:27 dist-test-slave-515x krb5kdc[31446](info): commencing operation
krb5kdc: starting...
W20260928 05:28:28.258112 8691 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/test-user.keytab.
Sep 28 05:28:28 dist-test-slave-515x krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.124.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
Entry for principal kudu/127.8.124.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab.
I20260928 05:28:28.512229 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46569
--webserver_interface=127.8.124.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:28:28.830457 31459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:28.830813 31459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:28.830915 31459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:28.831038 31459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:28.838333 31459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:28.838441 31459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:28.838526 31459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:28.838596 31459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:28.838670 31459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:28.848484 31459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.254.keytab
--principal=kudu/127.8.124.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.254:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.124.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:28.849735 31459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:28.851023 31459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:28.861981 31464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:28.861937 31467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:28.862017 31465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:28.863112 31459 server_base.cc:1034] running on GCE node
I20260928 05:28:28.863823 31459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:28.865159 31459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:28.866361 31459 hybrid_clock.cc:648] HybridClock initialized: now 1790573308866302 us; error 32 us; skew 500 ppm
I20260928 05:28:28.866662 31459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:28 dist-test-slave-515x krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:28.873430 31459 init.cc:377] Logged in from keytab as kudu/127.8.124.254@KRBTEST.COM (short username kudu)
I20260928 05:28:28.874861 31459 webserver.cc:460] Webserver started at http://127.8.124.254:39691/ using document root <none> and password file <none>
I20260928 05:28:28.875509 31459 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:28.875595 31459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:28.875895 31459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:28.878791 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data/instance:
uuid: "fc956db496534a36bb99c7ba9d4f7612"
format_stamp: "Formatted at 2026-09-28 05:28:28 on dist-test-slave-515x"
server_key: "4c692869bc93248506333fc9040d1e62"
server_key_iv: "f6c17a6ce2f46a54c43cdbaeb9545655"
server_key_version: "encryptionkey@0"
I20260928 05:28:28.879539 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance:
uuid: "fc956db496534a36bb99c7ba9d4f7612"
format_stamp: "Formatted at 2026-09-28 05:28:28 on dist-test-slave-515x"
server_key: "4c692869bc93248506333fc9040d1e62"
server_key_iv: "f6c17a6ce2f46a54c43cdbaeb9545655"
server_key_version: "encryptionkey@0"
I20260928 05:28:28.884460 31459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 05:28:28.887683 31474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:28.888906 31459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 05:28:28.889060 31459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal
uuid: "fc956db496534a36bb99c7ba9d4f7612"
format_stamp: "Formatted at 2026-09-28 05:28:28 on dist-test-slave-515x"
server_key: "4c692869bc93248506333fc9040d1e62"
server_key_iv: "f6c17a6ce2f46a54c43cdbaeb9545655"
server_key_version: "encryptionkey@0"
I20260928 05:28:28.889246 31459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:28.900906 31459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:28.904599 31459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:28.904830 31459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:28.921489 31459 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.254:46569
I20260928 05:28:28.921501 31525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.254:46569 every 8 connection(s)
I20260928 05:28:28.922791 31459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/data/info.pb
I20260928 05:28:28.925833 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31459
I20260928 05:28:28.925992 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-0/wal/instance
I20260928 05:28:28.926381 8691 external_mini_cluster.cc:1428] Setting key 6643024396b90eaf2c1915e32e273448
I20260928 05:28:28.926532 31526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:28.933539 31526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:28 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.124.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.253@KRBTEST.COM" created.
I20260928 05:28:28.957134 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:28.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.8.124.254:46569)
0928 05:28:28.937909 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:28.937922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:28.938510 (+ 588us) server_negotiation.cc:407] Connection header received
0928 05:28:28.940446 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:28.940453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:28.940747 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:28.940961 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:28.942128 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:28.943864 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:28.945415 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:28.946321 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:28.946520 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:28.949898 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:28.949906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:28.949919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:28.949948 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:28.952885 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:28.953745 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:28.953752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:28.953754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:28.953902 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:28.954326 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:28.954330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:28.954332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:28.954905 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:28.955108 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:28.955725 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 05:28:28.955979 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":426,"threads_started":1}
I20260928 05:28:28.957163 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:28.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46569 (local address 127.0.0.1:42722)
0928 05:28:28.938319 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:28.938377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:28.940094 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:28.940951 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:28.940957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:28.941517 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:28.941911 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:28.941931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:28.944039 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:28.944043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:28.945250 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:28.945266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:28.946542 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:28.946548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:28.946676 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:28.947322 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:28.947349 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:28.949658 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:28.953029 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:28.953042 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:28.953055 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:28.953570 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:28.954010 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:28.954015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:28.954018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:28.954203 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:28.955045 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:28.955052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:28.955320 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 05:28:28.955572 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 05:28:28.956157 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":97,"spinlock_wait_cycles":51584,"thread_start_us":775,"threads_started":1}
I20260928 05:28:28.960258 31526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:28.961362 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:28.960902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55398)
0928 05:28:28.961094 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:28.961221 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 05:28:28.961655 31478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
I20260928 05:28:28.963222 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:28.962835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55414)
0928 05:28:28.962987 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:28.963066 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:28:28.963438 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:45853: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
Entry for principal kudu/127.8.124.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab.
I20260928 05:28:28.984552 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:45853
--webserver_interface=127.8.124.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:28:29.012800 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } attempt: 1
I20260928 05:28:29.014017 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.013447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55430)
0928 05:28:29.013714 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.013854 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 05:28:29.015110 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.014687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55444)
0928 05:28:29.014826 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.014915 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:28:29.015339 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:45853: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
I20260928 05:28:29.083602 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } attempt: 2
I20260928 05:28:29.084894 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.084376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55454)
0928 05:28:29.084598 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.084726 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:28:29.085918 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.085508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55466)
0928 05:28:29.085690 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.085770 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:28:29.086097 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:45853: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
I20260928 05:28:29.177362 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } attempt: 3
I20260928 05:28:29.178572 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.178081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55470)
0928 05:28:29.178297 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.178409 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:28:29.179590 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.179201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55476)
0928 05:28:29.179345 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.179431 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 05:28:29.180006 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:45853: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
W20260928 05:28:29.307940 31533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:29.308255 31533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:29.308327 31533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:29.308405 31533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:29.315620 31533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:29.315707 31533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:29.315762 31533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:29.315810 31533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:29.315867 31533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:29.323264 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } attempt: 4
I20260928 05:28:29.324503 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.323916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55480)
0928 05:28:29.324226 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.324334 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 05:28:29.325474 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.325108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55492)
0928 05:28:29.325242 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.325331 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:28:29.325672 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.253:45853: Network error: Client connection negotiation failed: client connection to 127.8.124.253:45853: connect: Connection refused (error 111)
I20260928 05:28:29.326292 31533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.253.keytab
--principal=kudu/127.8.124.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.253:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.124.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:29.327523 31533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:29.328789 31533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:29.340216 31538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:29.340267 31541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:29.340216 31539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:29.340638 31533 server_base.cc:1034] running on GCE node
I20260928 05:28:29.341580 31533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:29.343026 31533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:29.344254 31533 hybrid_clock.cc:648] HybridClock initialized: now 1790573309344181 us; error 55 us; skew 500 ppm
I20260928 05:28:29.344583 31533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:29.347781 31533 init.cc:377] Logged in from keytab as kudu/127.8.124.253@KRBTEST.COM (short username kudu)
I20260928 05:28:29.349246 31533 webserver.cc:460] Webserver started at http://127.8.124.253:37847/ using document root <none> and password file <none>
I20260928 05:28:29.349975 31533 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:29.350076 31533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:29.350392 31533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:29.353292 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data/instance:
uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "84ff9536ef41f9f00ca5b27be538e1ef"
server_key_iv: "5cb5c584318a911fc90cbf589fcae8ec"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.354040 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal/instance:
uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "84ff9536ef41f9f00ca5b27be538e1ef"
server_key_iv: "5cb5c584318a911fc90cbf589fcae8ec"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.359018 31533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:28:29.362262 31548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:29.363605 31533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:28:29.363773 31533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal
uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "84ff9536ef41f9f00ca5b27be538e1ef"
server_key_iv: "5cb5c584318a911fc90cbf589fcae8ec"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.363963 31533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:29.405346 31533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:29.409921 31533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:29.410197 31533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:29.427099 31533 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.253:45853
I20260928 05:28:29.427119 31599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.253:45853 every 8 connection(s)
I20260928 05:28:29.428378 31533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/data/info.pb
I20260928 05:28:29.429031 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31533
I20260928 05:28:29.429199 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-1/wal/instance
I20260928 05:28:29.429759 8691 external_mini_cluster.cc:1428] Setting key aed5bf1cc56bd3da268f9851cf12cbc5
I20260928 05:28:29.432112 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:29.438498 31600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.124.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.124.252@KRBTEST.COM" created.
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:29.459865 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.441033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.8.124.254:46569)
0928 05:28:29.441239 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.441245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.442488 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 05:28:29.444347 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.444353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.444434 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.444545 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.446147 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.447168 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.448768 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.449395 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.449527 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.454359 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.454364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.454367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.454403 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.456854 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.457666 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.457672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.457675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.457822 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.458187 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.458192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.458196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.458531 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.458675 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.459325 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.459552 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:28:29.460650 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.441532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46569 (local address 127.0.0.1:42726)
0928 05:28:29.442316 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.442366 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.444093 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.444809 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.444817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.445371 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.445976 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.445995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.447354 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.447359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.448588 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.448602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.449649 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.449658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.449891 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.450558 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.450590 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.454094 (+ 3504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.456987 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.456996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.457014 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.457490 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.457915 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.457919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.457921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.458085 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.458637 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.458642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.458986 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.459202 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.459577 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":317,"threads_started":1}
I20260928 05:28:29.462337 31600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:29.475449 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.462922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55500)
0928 05:28:29.463084 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.463100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.463182 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.464149 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.464154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.464560 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.464693 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.464701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.466031 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.466037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.466850 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.466861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.467673 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.467679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.467834 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.468353 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.468369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.470253 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.473230 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.473235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.473240 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.473553 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.473892 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.473897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.473901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.474024 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.474568 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.474572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.474709 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.474823 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.475073 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:28:29.475558 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.463165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.8.124.253:45853)
0928 05:28:29.463770 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.463775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.463800 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:28:29.463839 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.463845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.464015 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.464193 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.464847 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.465910 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.466973 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.467540 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.467652 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.470432 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.470443 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.470455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.470485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.473114 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.473683 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.473690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.473693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.473797 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.474114 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.474119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.474122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.474438 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.474631 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.474986 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.475196 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
I20260928 05:28:29.477829 31600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:29.479094 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.478599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46810)
0928 05:28:29.478808 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.478913 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 05:28:29.479360 31550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.480751 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.480413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46814)
0928 05:28:29.480572 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.480649 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 05:28:29.480942 31600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
Entry for principal kudu/127.8.124.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab.
I20260928 05:28:29.483035 8691 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
/tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:33475
--webserver_interface=127.8.124.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.124.212:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:28:29.530282 31600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 1
I20260928 05:28:29.531448 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.531002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46826)
0928 05:28:29.531190 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.531302 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:28:29.532459 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.532135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46842)
0928 05:28:29.532266 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.532354 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 05:28:29.532678 31600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.600879 31600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 2
I20260928 05:28:29.602065 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.601625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46852)
0928 05:28:29.601824 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.601913 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 05:28:29.603053 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.602748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46854)
0928 05:28:29.602877 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.602945 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:28:29.603267 31600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.624919 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } attempt: 5
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:29.639421 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.625755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.8.124.253:45853)
0928 05:28:29.625922 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.625927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.625986 (+ 59us) server_negotiation.cc:407] Connection header received
0928 05:28:29.626164 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.626167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.626244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.626350 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.627307 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.627980 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.629732 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.630239 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.630349 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.634168 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.634173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.634177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.634211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.636747 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.637547 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.637551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.637554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.637712 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.638111 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.638114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.638116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.638402 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.638510 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.638995 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.639178 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:28:29.639565 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.625656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55510)
0928 05:28:29.625878 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.625901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.626053 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.626421 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.626426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.626906 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.627149 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.627164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.628118 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.628125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.629502 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.629520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.630433 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.630439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.630632 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.631259 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.631283 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.633949 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.636889 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.636895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.636899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.637416 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.637815 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.637820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.637824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.638007 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.638519 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.638524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.638705 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.638873 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.639182 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:28:29.640717 31526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:29.642019 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.641502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46864)
0928 05:28:29.641735 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.641839 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:28:29.643066 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.642669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46878)
0928 05:28:29.642834 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.642938 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 05:28:29.643276 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.694463 31600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 3
I20260928 05:28:29.694533 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 1
I20260928 05:28:29.695684 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.695122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46888)
0928 05:28:29.695431 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.695514 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260928 05:28:29.695685 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.695122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46892)
0928 05:28:29.695377 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.695491 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:28:29.696741 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.696377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46894)
0928 05:28:29.696535 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.696619 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:28:29.696857 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.696425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46902)
0928 05:28:29.696631 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.696724 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 05:28:29.696954 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
W20260928 05:28:29.697060 31600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.765198 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 2
I20260928 05:28:29.766448 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.765899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46908)
0928 05:28:29.766133 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.766249 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:28:29.767457 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.767082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46916)
0928 05:28:29.767219 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.767322 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:28:29.767652 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
W20260928 05:28:29.806041 31607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:28:29.806361 31607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:28:29.806432 31607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:28:29.806511 31607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:28:29.813796 31607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:28:29.813898 31607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:28:29.813953 31607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:28:29.814002 31607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:28:29.814061 31607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:28:29.823853 31607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.124.212:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.124.254:46569,127.8.124.253:45853,127.8.124.252:33475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/krb5kdc/kudu_127.8.124.252.keytab
--principal=kudu/127.8.124.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.124.252:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.124.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:28:29.825088 31607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:28:29.826395 31607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:28:29.838012 31612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:29.837982 31615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:28:29.837953 31613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:28:29.839143 31607 server_base.cc:1034] running on GCE node
I20260928 05:28:29.839911 31607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:28:29.840271 31600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 4
I20260928 05:28:29.841447 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.840903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46918)
0928 05:28:29.841146 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.841256 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 05:28:29.841748 31607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:28:29.842659 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.842309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46922)
0928 05:28:29.842468 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.842539 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:28:29.842867 31600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.842988 31607 hybrid_clock.cc:648] HybridClock initialized: now 1790573309842894 us; error 61 us; skew 500 ppm
I20260928 05:28:29.843297 31607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:28:29.846395 31607 init.cc:377] Logged in from keytab as kudu/127.8.124.252@KRBTEST.COM (short username kudu)
I20260928 05:28:29.847766 31607 webserver.cc:460] Webserver started at http://127.8.124.252:39241/ using document root <none> and password file <none>
I20260928 05:28:29.848414 31607 fs_manager.cc:359] Metadata directory not provided
I20260928 05:28:29.848498 31607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:28:29.848771 31607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:28:29.851720 31607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data/instance:
uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "e751d307ac3a609a6fd57765f1b1bdef"
server_key_iv: "666ccd5b87e26e88b05146391274982e"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.852448 31607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal/instance:
uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "e751d307ac3a609a6fd57765f1b1bdef"
server_key_iv: "666ccd5b87e26e88b05146391274982e"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.857726 31607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:28:29.861881 31622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:28:29.863456 31607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:28:29.863622 31607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data,/tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal
uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c"
format_stamp: "Formatted at 2026-09-28 05:28:29 on dist-test-slave-515x"
server_key: "e751d307ac3a609a6fd57765f1b1bdef"
server_key_iv: "666ccd5b87e26e88b05146391274982e"
server_key_version: "encryptionkey@0"
I20260928 05:28:29.863821 31607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:28:29.873832 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 3
I20260928 05:28:29.874913 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.874447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46928)
0928 05:28:29.874677 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.874766 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 05:28:29.875909 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.875545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46932)
0928 05:28:29.875726 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.875806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 05:28:29.876118 31526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.124.252:33475: Network error: Client connection negotiation failed: client connection to 127.8.124.252:33475: connect: Connection refused (error 111)
I20260928 05:28:29.884411 31607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:28:29.888569 31607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:28:29.888795 31607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:28:29.906596 31607 rpc_server.cc:307] RPC server started. Bound to: 127.8.124.252:33475
I20260928 05:28:29.906615 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.124.252:33475 every 8 connection(s)
I20260928 05:28:29.907977 31607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/data/info.pb
I20260928 05:28:29.911754 31674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:28:29.916615 8691 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu as pid 31607
I20260928 05:28:29.916805 8691 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbMG2Xa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572965871275-8691-0/minicluster-data/master-2/wal/instance
I20260928 05:28:29.917327 8691 external_mini_cluster.cc:1428] Setting key cd7bf92d86104ab045ff5d4fdb9b97c5
I20260928 05:28:29.918983 31674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:28:29.940644 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.921955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.8.124.254:46569)
0928 05:28:29.922154 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.922159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.924456 (+ 2297us) server_negotiation.cc:407] Connection header received
0928 05:28:29.924702 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.924708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.924777 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.924866 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.926247 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.926969 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.929044 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.929502 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.929638 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.933968 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.933973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.933977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.934014 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.937094 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.938014 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.938020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.938024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.938167 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.938684 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.938691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.938694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.939078 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.939218 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.940035 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.940272 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:28:29.941752 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.919236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.8.124.252:33475)
0928 05:28:29.920001 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.920013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.920056 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:28:29.922281 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.922312 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.922873 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.923313 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.925023 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.927105 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.928904 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.929809 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.930048 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.934466 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.934477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.934496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.934533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.937926 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.938775 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.938784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.938787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.938934 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.939397 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.939402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.939404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.939808 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.940099 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.940408 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.940638 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":564,"spinlock_wait_cycles":27776,"thread_start_us":357,"threads_started":1}
I20260928 05:28:29.941807 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.921961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.254:46569 (local address 127.0.0.1:42736)
0928 05:28:29.924324 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.924370 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.924568 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.924946 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.924952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.925676 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.926071 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.926087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.927132 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.927139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.928908 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.928922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.929788 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.929797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.930045 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.930990 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.931020 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.933727 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.937264 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.937276 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.937287 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.937884 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.938283 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.938288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.938292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.938523 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.939220 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.939228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.939609 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.939886 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.940403 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"mutex_wait_us":49,"thread_start_us":1988,"threads_started":1}
I20260928 05:28:29.943317 31674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.254@KRBTEST.COM
I20260928 05:28:29.957031 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.943955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.253:45853 (local address 127.0.0.1:55526)
0928 05:28:29.944201 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.944219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.944317 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.944670 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.944675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.945112 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.945281 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.945292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.946254 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.946259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.947218 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.947233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.947911 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.947917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.948137 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.948694 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.948711 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.951191 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.954569 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.954574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.954576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.954991 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.955408 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.955413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.955416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.955592 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.956124 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.956129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.956298 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.956454 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.956742 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 05:28:29.957123 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.944109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.8.124.253:45853)
0928 05:28:29.944408 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.944414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.944431 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:28:29.944483 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.944487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.944543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.944621 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.945404 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.946117 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.947367 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.947776 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.947867 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.951400 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.951406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.951409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.951444 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.954445 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.955093 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.955099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.955102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.955268 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.955694 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.955698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.955702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.956004 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.956132 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.956570 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.956766 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 05:28:29.957978 31674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:28:29.959914 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.944800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.8.124.254:46569)
0928 05:28:29.944940 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.944944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.944959 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:28:29.945118 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.945122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.945178 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.945248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.946488 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.947217 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.948634 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.949056 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.949147 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.953515 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.953523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.953528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.953570 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.956721 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.957586 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.957592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.957634 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.957791 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.958199 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.958204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.958207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.958565 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.958833 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.959372 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.959584 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.252@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
Sep 28 05:28:29 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.124.253@KRBTEST.COM
I20260928 05:28:29.970899 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.958765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46944)
0928 05:28:29.958968 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:29.958981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:29.959061 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:29.959338 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:29.959343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:29.959783 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:29.959952 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.959964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.961034 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.961039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.962145 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:29.962159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.962815 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.962821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:29.963035 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.963605 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:29.963621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:29.965679 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:29.968765 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.968770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.968774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.969091 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.969412 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:29.969416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:29.969420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:29.969552 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:29.970112 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:29.970118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:29.970292 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 05:28:29.970428 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 05:28:29.970626 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 05:28:29.971199 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.958742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.8.124.252:33475)
0928 05:28:29.958921 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.958927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.959034 (+ 107us) server_negotiation.cc:407] Connection header received
0928 05:28:29.959136 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.959142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.959217 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.959301 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.960093 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.960927 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.962281 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.962667 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.962760 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.965847 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.965851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.965854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.965880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.968630 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.969190 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.969195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.969199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.969312 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.969700 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.969704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.969706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.969991 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.970100 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.970585 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.970816 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:28:29.975947 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:29.961581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.8.124.253:45853)
0928 05:28:29.961752 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:29.961757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:29.961772 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:28:29.961867 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:29.961871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:29.961923 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:29.961994 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:29.963346 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.964104 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.965556 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:29.966004 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:29.966117 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:29.970385 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:29.970390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:29.970394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:29.970429 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:29.973520 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.974174 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.974180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.974183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.974339 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:29.974665 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:29.974670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:29.974673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:29.974988 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:29.975242 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:29.975417 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 05:28:29.975637 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:28:29.976848 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: Bootstrap starting.
I20260928 05:28:29.980515 8691 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:28:29.980634 31674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:29.981904 31674 log.cc:824] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:29.984557 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: No bootstrap required, opened a new log
I20260928 05:28:29.983407 31501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fc956db496534a36bb99c7ba9d4f7612"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42742
I20260928 05:28:29.991492 31674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:29.991783 31674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea7bc6abe94f3abd255c31b6e9fe2c, State: Initialized, Role: FOLLOWER
I20260928 05:28:29.992254 31674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:29.993076 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:29.993462 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:29.994182 31674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:28:30.002872 31691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:28:30.002990 31691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:28:30.001022 31607 master_runner.cc:186] Error getting master registration for 127.8.124.254:46569: OK, instance_id { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" instance_seqno: 1790573308916791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:28:30.011934 31607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:28:30.053383 31526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 4
Sep 28 05:28:30 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.254@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
I20260928 05:28:30.070056 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.054136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46958)
0928 05:28:30.054342 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:30.054363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:30.054472 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:30.054850 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:30.054855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:30.055359 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:30.055559 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:30.055572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.057015 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.057021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:30.058513 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:30.058529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.059550 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.059557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:30.059777 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.060365 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:30.060390 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:30.063197 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:30.067096 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:30.067102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:30.067104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:30.067640 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:30.068012 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:30.068017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:30.068021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:30.068175 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:30.069019 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:30.069025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:30.069195 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 05:28:30.069339 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:28:30.069651 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:28:30.071435 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.054210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.8.124.252:33475)
0928 05:28:30.054427 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.054434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.054456 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:28:30.054592 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.054598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.054679 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.054803 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.055734 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.056876 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.058698 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.059402 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.059515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.063386 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:30.063393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:30.063398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:30.063442 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:30.066958 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.067755 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.067761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.067765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.067907 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.068283 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.068288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.068292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.068888 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:30.069057 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:30.070837 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0928 05:28:30.071070 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:28:30.077036 31526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: Bootstrap starting.
I20260928 05:28:30.080875 31526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:30.082170 31526 log.cc:824] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:30.085862 31526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: No bootstrap required, opened a new log
I20260928 05:28:30.093107 31526 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.093487 31526 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc956db496534a36bb99c7ba9d4f7612, State: Initialized, Role: FOLLOWER
I20260928 05:28:30.093984 31526 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.095147 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.095712 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:30.096334 31526 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:30.096379 31501 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:28:30.096551 31501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:30.099977 31501 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.102902 31501 leader_election.cc:288] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [CANDIDATE]: Term 1 election: Requested vote from peers baa4ba4915b34a6a8f14abfdc0a8c45f (127.8.124.253:45853), 14ea7bc6abe94f3abd255c31b6e9fe2c (127.8.124.252:33475)
I20260928 05:28:30.104490 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc956db496534a36bb99c7ba9d4f7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c"
I20260928 05:28:30.105022 31649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:30.108770 31649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc956db496534a36bb99c7ba9d4f7612 in term 1.
I20260928 05:28:30.109506 31476 leader_election.cc:302] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea7bc6abe94f3abd255c31b6e9fe2c, fc956db496534a36bb99c7ba9d4f7612; no voters:
I20260928 05:28:30.110018 31694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:28:30.121271 31694 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [term 1 LEADER]: Becoming Leader. State: Replica: fc956db496534a36bb99c7ba9d4f7612, State: Running, Role: LEADER
I20260928 05:28:30.122480 31694 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.130765 31702 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc956db496534a36bb99c7ba9d4f7612. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.131510 31702 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:30.133466 31704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 05:28:30.140899 31628 tablet.cc:2369] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:28:30.141153 31704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:28:30.142057 31600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } attempt: 5
Sep 28 05:28:30 dist-test-slave-515x krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790573309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.124.253@KRBTEST.COM for kudu/127.8.124.252@KRBTEST.COM
W20260928 05:28:30.157680 31480 tablet.cc:2369] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:28:30.166952 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.142723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.124.252:33475 (local address 127.0.0.1:46962)
0928 05:28:30.143102 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 05:28:30.143123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:28:30.143249 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:28:30.144172 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:28:30.144179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:28:30.144712 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:28:30.144942 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:30.144956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.146710 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.146716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:30.148099 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:28:30.148116 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.149430 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.149435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:28:30.149722 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.150650 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:28:30.150697 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:28:30.154410 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:28:30.161498 (+ 7088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:30.161525 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:30.161530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:30.162496 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:30.163313 (+ 817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:28:30.163319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:28:30.163324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:28:30.163606 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:28:30.164644 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:28:30.164650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:28:30.164900 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 05:28:30.165105 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 05:28:30.165570 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 05:28:30.168394 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.143473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.8.124.252:33475)
0928 05:28:30.143637 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.143643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.143664 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:28:30.143732 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.143742 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.143821 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.143921 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.145193 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.146509 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.148380 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.149077 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.149205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.154688 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:30.154695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:30.154699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:30.154740 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:30.161268 (+ 6528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.162780 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.162786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.162815 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.163095 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.163819 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.163825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.163828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.164467 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:30.165866 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:30.166368 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 05:28:30.167221 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 05:28:30.169780 31714 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:28:30.178524 31649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [term 1 FOLLOWER]: Refusing update from remote peer fc956db496534a36bb99c7ba9d4f7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:28:30.179874 31694 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:28:30.191800 31600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: Bootstrap starting.
I20260928 05:28:30.209924 31600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: Neither blocks nor log segments found. Creating new log.
I20260928 05:28:30.211553 31600 log.cc:824] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:28:30.213294 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc956db496534a36bb99c7ba9d4f7612. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.213661 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:30.216548 31710 mvcc.cc:205] Tried to move back new op lower bound from 7334188278461304832 to 7334188278272622592. Current Snapshot: MvccSnapshot[applied={T|T < 7334188278461304832}]
I20260928 05:28:30.218407 31718 mvcc.cc:205] Tried to move back new op lower bound from 7334188278461304832 to 7334188278272622592. Current Snapshot: MvccSnapshot[applied={T|T < 7334188278461304832}]
I20260928 05:28:30.219112 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.219398 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: This master's current role is: LEADER
I20260928 05:28:30.219969 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.220227 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:30.223554 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.183069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.8.124.254:46569)
0928 05:28:30.185775 (+ 2706us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.185782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.198683 (+ 12901us) server_negotiation.cc:407] Connection header received
0928 05:28:30.199726 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.199732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.199819 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.199894 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.203397 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.204733 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.206081 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.206880 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.207011 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.210872 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:30.210880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:30.210886 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:30.210940 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:30.219081 (+ 8141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.220268 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.220307 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.220310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.220483 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.220988 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.220993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.220997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.221383 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:30.221673 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:30.222939 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0928 05:28:30.223179 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2574}
I20260928 05:28:30.223914 31704 catalog_manager.cc:1347] Generated new cluster ID: 1390de639e0c4061b1c885ed853bacf7
I20260928 05:28:30.224007 31704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:28:30.183043 31459 master_runner.cc:186] Error getting master registration for 127.8.124.253:45853: OK, instance_id { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" instance_seqno: 1790573309422326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:28:30.234628 31459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:28:30.237985 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.185915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.8.124.252:33475)
0928 05:28:30.197954 (+ 12039us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.197962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.214954 (+ 16992us) server_negotiation.cc:407] Connection header received
0928 05:28:30.220966 (+ 6012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.220973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.221059 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.221134 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.223332 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.224597 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.226593 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.227251 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.227344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.230713 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:30.230721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:30.230726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:30.230766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:30.234378 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.235551 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.235557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.235560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.235728 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.236181 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.236187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.236190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.236535 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:30.236795 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:30.237320 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 05:28:30.237548 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11611,"thread_start_us":11033,"threads_started":1}
I20260928 05:28:30.238144 31600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: No bootstrap required, opened a new log
I20260928 05:28:30.253842 31704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:28:30.255628 31704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:28:30.260601 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:30.215063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.8.124.253:45853)
0928 05:28:30.215234 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.215239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.222736 (+ 7497us) server_negotiation.cc:407] Connection header received
0928 05:28:30.224332 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.224338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.224420 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.224491 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.234868 (+ 10377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.235924 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.249110 (+ 13186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:30.249788 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:30.249915 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:30.253326 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:28:30.253334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:28:30.253339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:28:30.253382 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:28:30.256934 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.257844 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.257850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.257853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.258003 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:28:30.258406 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:28:30.258411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:28:30.258415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:28:30.258762 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:28:30.258986 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:30.260035 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0928 05:28:30.260260 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:28:30.249007 31600 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.262516 31600 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa4ba4915b34a6a8f14abfdc0a8c45f, State: Initialized, Role: FOLLOWER
I20260928 05:28:30.263171 31600 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } }
I20260928 05:28:30.269584 31704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612: Generated new TSK 0
I20260928 05:28:30.270717 31704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:28:30.274507 31724 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.275278 31724 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:30.275712 31600 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:28:30.276093 31574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:28:30.279999 31574 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [term 1 FOLLOWER]: Refusing update from remote peer fc956db496534a36bb99c7ba9d4f7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:28:30.281069 31702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:28:30.285538 31575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc956db496534a36bb99c7ba9d4f7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f"
I20260928 05:28:30.289458 31575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc956db496534a36bb99c7ba9d4f7612 in term 1.
I20260928 05:28:30.359253 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.359712 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:28:30.359248 31724 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc956db496534a36bb99c7ba9d4f7612. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.360199 31724 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 05:28:30.403578 8691 mini_kdc.cc:348] Time spent kdestroy: real 0.107s user 0.002s sys 0.023s
I20260928 05:28:30.416678 31743 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 05:28:30.422029 31717 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.419825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.8.124.252:33475)
0928 05:28:30.419997 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.420002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.420025 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:28:30.420271 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.420276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.420397 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.420471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.421711 (+ 1240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46982: BlockingRecv error: recv got EOF from 127.0.0.1:46982 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 05:28:30.424144 31529 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.420346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.8.124.254:46569)
0928 05:28:30.420524 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.420529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.422568 (+ 2039us) server_negotiation.cc:407] Connection header received
0928 05:28:30.422797 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.422804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.422896 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.422969 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.423929 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42762: BlockingRecv error: recv got EOF from 127.0.0.1:42762 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 05:28:30.426365 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.424589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.8.124.253:45853)
0928 05:28:30.424760 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.424766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.424982 (+ 216us) server_negotiation.cc:407] Connection header received
0928 05:28:30.425194 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.425199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.425281 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.425344 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.426154 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55556: BlockingRecv error: recv got EOF from 127.0.0.1:55556 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 05:28:30.460726 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.458235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.8.124.253:45853)
0928 05:28:30.458440 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.458446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.459078 (+ 632us) server_negotiation.cc:407] Connection header received
0928 05:28:30.459324 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.459329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.459456 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.459556 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.460461 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55568: BlockingRecv error: recv got EOF from 127.0.0.1:55568 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 05:28:30.471516 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.467464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.8.124.253:45853)
0928 05:28:30.467648 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.467653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.470105 (+ 2452us) server_negotiation.cc:407] Connection header received
0928 05:28:30.470334 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.470340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.470429 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.470526 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.471306 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55578: BlockingRecv error: recv got EOF from 127.0.0.1:55578 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 05:28:30.484450 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.478932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.8.124.253:45853)
0928 05:28:30.479089 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.479096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.482130 (+ 3034us) server_negotiation.cc:407] Connection header received
0928 05:28:30.482324 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.482330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.482428 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.482532 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.484212 (+ 1680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55590: BlockingRecv error: recv got EOF from 127.0.0.1:55590 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 05:28:30.496642 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.494410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.8.124.253:45853)
0928 05:28:30.494599 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.494605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.495086 (+ 481us) server_negotiation.cc:407] Connection header received
0928 05:28:30.495340 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.495346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.495438 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.495541 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.496418 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55606: BlockingRecv error: recv got EOF from 127.0.0.1:55606 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 05:28:30.510252 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.507967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.8.124.253:45853)
0928 05:28:30.508153 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.508160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.508778 (+ 618us) server_negotiation.cc:407] Connection header received
0928 05:28:30.508985 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.508991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.509081 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.509187 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.510002 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55614: BlockingRecv error: recv got EOF from 127.0.0.1:55614 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 05:28:30.531874 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.529767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.8.124.253:45853)
0928 05:28:30.529992 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.529998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.530337 (+ 339us) server_negotiation.cc:407] Connection header received
0928 05:28:30.530577 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.530581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.530663 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.530764 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.531640 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55622: BlockingRecv error: recv got EOF from 127.0.0.1:55622 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 05:28:30.570091 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.567962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.8.124.253:45853)
0928 05:28:30.568216 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.568222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.568708 (+ 486us) server_negotiation.cc:407] Connection header received
0928 05:28:30.568899 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.568906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.569002 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.569117 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.569831 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55636: BlockingRecv error: recv got EOF from 127.0.0.1:55636 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260928 05:28:30.639637 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.637713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.8.124.253:45853)
0928 05:28:30.637946 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.637971 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.638247 (+ 276us) server_negotiation.cc:407] Connection header received
0928 05:28:30.638433 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.638437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.638509 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.638594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.639431 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55646: BlockingRecv error: recv got EOF from 127.0.0.1:55646 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:28:30.714111 31702 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc956db496534a36bb99c7ba9d4f7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc956db496534a36bb99c7ba9d4f7612" member_type: VOTER last_known_addr { host: "127.8.124.254" port: 46569 } } peers { permanent_uuid: "baa4ba4915b34a6a8f14abfdc0a8c45f" member_type: VOTER last_known_addr { host: "127.8.124.253" port: 45853 } } peers { permanent_uuid: "14ea7bc6abe94f3abd255c31b6e9fe2c" member_type: VOTER last_known_addr { host: "127.8.124.252" port: 33475 } } }
I20260928 05:28:30.714522 31702 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc956db496534a36bb99c7ba9d4f7612 [sys.catalog]: This master's current role is: LEADER
W20260928 05:28:30.777187 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:30.775218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.8.124.253:45853)
0928 05:28:30.775409 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:30.775415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:30.775819 (+ 404us) server_negotiation.cc:407] Connection header received
0928 05:28:30.776026 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:30.776030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:30.776103 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:30.776191 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:30.776987 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55658: BlockingRecv error: recv got EOF from 127.0.0.1:55658 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:28:31.005000 31691 catalog_manager.cc:1259] Loaded cluster ID: 1390de639e0c4061b1c885ed853bacf7
I20260928 05:28:31.005111 31691 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: loading cluster ID for follower catalog manager: success
I20260928 05:28:31.007608 31691 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: acquiring CA information for follower catalog manager: success
I20260928 05:28:31.009349 31691 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14ea7bc6abe94f3abd255c31b6e9fe2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:28:31.038491 31605 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:28:31.036921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.8.124.253:45853)
0928 05:28:31.037137 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.037143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.037284 (+ 141us) server_negotiation.cc:407] Connection header received
0928 05:28:31.037481 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.037485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.037559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.037677 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:28:31.038282 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55670: BlockingRecv error: recv got EOF from 127.0.0.1:55670 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 05:28:31.420776 31743 catalog_manager.cc:1259] Loaded cluster ID: 1390de639e0c4061b1c885ed853bacf7
I20260928 05:28:31.420902 31743 catalog_manager.cc:1552] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: loading cluster ID for follower catalog manager: success
I20260928 05:28:31.423859 31743 catalog_manager.cc:1574] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: acquiring CA information for follower catalog manager: success
I20260928 05:28:31.425559 31743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P baa4ba4915b34a6a8f14abfdc0a8c45f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:28:31.560786 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:31.554039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.8.124.253:45853)
0928 05:28:31.554802 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.554807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.554824 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:28:31.554887 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.554891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.555042 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.555146 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:31.555652 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.556456 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.558417 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.558888 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.559000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:31.559581 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:31.560162 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:31.560285 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:31.560408 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 05:28:31.560477 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":487,"threads_started":1}
I20260928 05:28:31.581799 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:31.566949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.8.124.252:33475)
0928 05:28:31.576147 (+ 9198us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.576151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.576169 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:28:31.576272 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.576277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.576425 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.576563 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:31.577056 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.577886 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.579628 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.580027 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.580118 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:31.580668 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:31.581158 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:31.581308 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:31.581421 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 05:28:31.581489 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9073,"thread_start_us":8979,"threads_started":1}
I20260928 05:28:31.607385 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:31.588152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.8.124.252:33475)
0928 05:28:31.588310 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.588315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.600976 (+ 12661us) server_negotiation.cc:407] Connection header received
0928 05:28:31.601206 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.601212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.601293 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.601346 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:31.602138 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.603164 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.605243 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.605862 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.605955 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:31.606598 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:31.606734 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:31.606794 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:31.607030 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 05:28:31.607105 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:28:31.608768 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:31.587597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.8.124.254:46569)
0928 05:28:31.600003 (+ 12406us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.600010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.600879 (+ 869us) server_negotiation.cc:407] Connection header received
0928 05:28:31.601246 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.601252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.601468 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.601662 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:31.602490 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.603634 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.606010 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.606684 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.606807 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:31.607513 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:31.607923 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:31.608022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:31.608409 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 05:28:31.608491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12252,"thread_start_us":12790,"threads_started":1}
I20260928 05:28:31.612818 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31459
I20260928 05:28:31.613852 31521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:31.614925 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:28:31.601085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.8.124.253:45853)
0928 05:28:31.601269 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:28:31.601275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:28:31.608015 (+ 6740us) server_negotiation.cc:407] Connection header received
0928 05:28:31.609288 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:28:31.609294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:28:31.609383 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:28:31.609473 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:28:31.610018 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.611082 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.613020 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:28:31.613654 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:28:31.613776 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:28:31.614238 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:28:31.614360 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:28:31.614410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 05:28:31.614646 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 05:28:31.614729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:28:31.767285 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31459
W20260928 05:28:31.783805 31626 connection.cc:535] server connection from 127.0.0.1:46958 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 05:28:31.784128 31549 connection.cc:535] server connection from 127.0.0.1:55510 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 05:28:31.784446 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31533
I20260928 05:28:31.785481 31595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:31.894107 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31533
I20260928 05:28:31.910519 8691 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu pid 31607
I20260928 05:28:31.911577 31669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:28:32.015990 8691 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbMG2Xa/build/asan/bin/kudu with pid 31607
2026-09-28T05:28:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4782 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9170 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346110 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST